By David Maggs, Metcalf Fellow on Arts and Society

If you happened to catch Michelle Chawla and Lise Ann Johnston from the Canada Council discussing federal arts funding on a recent Business / Arts webinar, you know the mood in Canada’s cultural sector isn’t bursting with optimism. Facing a challenging future, they mentioned two themes Canada’s cultural sector might want to address. First, strengthening a common purpose and voice to present a clear, compelling value proposition to the country, and second, showing this value proposition actually matters to Canadians and that we are, indeed, delivering on its promise. In other words, a good, clear story along with the data to back it up.

These themes have been top of mind as I conclude this series of discussions on art and social impact. In conversations with Darren O’Donnell and Nina Lee Aquino, we saw two artists with strong social agendas. While one works far outside traditional forms and the other leads a major institution, they share an aesthetic focus as their path to impact — art that does good needs to be good. But they also share a question: “What exactly is the social impact of that? I don’t know.”

Aye, as someone once said, there’s the rub. Two internationally-active Canadian artists doing groundbreaking work to great acclaim, and yet this doesn’t give Michelle and Lise Ann what they need to defend arts funding in the current fiscal landscape. Absent here is evaluation. The task of generating qualitative and quantitative data around the impact of that activity. If our discussions of art and social impact illustrated compelling visions for bringing these worlds together, without evaluation we aren’t helping Michelle, Lise Ann, or anyone else trying to keep cultural budgets from withering in the face of austerity. For this, I needed to talk to Robin Sokoloski from Mass Culture.

“That’s why Mass Culture exists,” she begins. “A group of us running national arts service organizations, going to Parliament every year to ask for money without any research or data backing us up. It was frustrating. It felt empty, and it didn’t allow us to come together as an arts community and ask for things that were relevant to us, things that met more specific needs than just ‘money please.’”

“And that’s how you got interested in evaluation?” I ask.

“We’re trying to build the capacity for arts organizations to better understand what their artistic experiences are — and are not — doing out there in the world,” she replies.

I get why funders are after this, as it provides clear, comparable defences against budget cuts, “but what about organizations?” I wonder, “why do they want a scorecard at the end of a production?”

This earns me a scowl. “That’s evaluation at its worst,” Sokoloski grumbles. “Evaluation should never be an endpoint. It needs to be integrated into programming, development, outreach — whatever’s happening inside an organization. Way too often we put effort into these activities and then have no idea if they work. Was it time and money well spent? Do we even have the capacity to deliver on that vision? With evaluation, we can set learning points around an initiative, identify the intentions of what we are doing, structure the processes we need to clarify and optimize our activity, and understand if it mattered to people in the ways we intended. So before it’s for a funder, it’s for us. Are we doing what we say we are doing? What are our shortcomings? What needs do those shortcomings reveal? This sort of work isn’t happening nearly enough in the sector.”

“That sounds so different than tracking ticket sales and artists served, or trying to calculate economic impact,” I observe.

“Exactly, and we know the pressure on the sector to produce that kind of data,” says Sokoloski, “but we are not showing society what we have to offer if that’s all we ever manage to say about ourselves.”

“But isn’t it anecdotes after that?” I question, “I mean, isn’t that what Michelle was lamenting at the webinar — that we need to get our data beyond anecdotes?”

“And we can if we have the frameworks in place to structure how those anecdotes are collected and accounted for,” Sokoloski explains. “Right now, Mass Culture is collaborating on a variety of initiatives to improve our ability to generate data that gets beyond those easily quantifiable aspects.”

I lean towards my webcam and use my “give me an example” face.

“Animating Democracy’s Aesthetic Perspectives initiative was built in conjunction with evaluators and artists,” Sokoloski explains. “It offers eleven attributes to understand the impact of an artistic experience. One of my favorites is ‘stickiness,’ understanding the sustained resonance of experience. It’s an approach that gets us beyond standard quantitative analyses to target impacts on things like, say, ‘hopefulness.’ This approach to evaluation gives us a means to establish the way certain artistic experiences create a sense of hope.”

“Or don’t?” I suggest.

“Or don’t,” Sokoloski nods, “that’s why we need to avoid that scorecard approach. It has to be an iterative, adaptive relationship. Organizations have to be willing to step back and learn, understand whether or not this is having that amazing impact they think it is. We have to be open to the fact that it might not be amazing, right?” Sokoloski adds thoughtfully, “which might be a source of resistance right there.”

“That this isn’t just an affirming exercise?” I ask.

“We can’t develop this evaluative process for the purposes of advocacy, then it would just be propaganda. So maybe that’s why we are hesitant to go down this road? It might tell us something we don’t want to know? That the arts aren’t always as impactful as we think they are?”

We both get a little quiet. “But…” I begin hesitantly, “we do want to know that, don’t we?”

“We need to know,” Sokoloski is emphatic. “The rhetoric, no matter how grand, just doesn’t cut it anymore. We need to know if we are having the impact we think we’re having, so that everybody else knows it when we are, and that we know it when we aren’t.”

“Are we coaxing the ostrich here?” I ask.

“Maybe,” she shrugs, “I’ve often said to the people we work with that we might show you something you aren’t going to be happy seeing, and that needs to be okay. So if they think it’s a scorecard, who wouldn’t plunk their heads in the sand? But if we approach it as empowering change? Something that gives us insight to improve?”

I can see how this spirit of evaluation might get us beyond the empty advocacy Sokoloski was lamenting, that it can demonstrate the impact we are having, and help us move towards our goals with greater precision. But what about the goals themselves? “What if the goals are the problem?” I wonder aloud, “what if they don’t matter to people in the way that we think they do?”

“I’m fascinated with how best to engage community,” Sokoloski explains. “I have often used this process of developing evaluation frameworks as a tool to help organizations look beyond themselves, beyond the sector. We need to include voices from outside of the arts in developing these frameworks, so that they are telling us what would matter to them, and what they would want to know through an evaluation process. If we do that, then we aren’t just optimizing for our own needs, we are incorporating the cultural needs of those beyond our typical scope as well.”

This use of evaluation seems to support the growing instinct in the sector to turn outwards, as described by Manifesto for Now. I think of this as a potential Copernican moment for the arts, a shift from an artist-centred universe to an audience-centred universe. Could embedding some of this reorientation in evaluation frameworks make for a smoother ride, as we brave the work to wind up somewhere new?

“Data-shmata, Robin,” I interject, with sudden skepticism. I am remembering my conversation with Patti Pon about our recent pandemic exit strategy. “We had all the data we needed to tell us audiences weren’t dying to jump back into live venues with us and we ignored it. We plowed cash into the sector, over-produced, under-sold, and blew once-in-a-generation cultural funding on a post-pandemic nostalgia party instead.”

“Instead of what?” she asks.

“Instead of saying, ‘hey, look at this data, it looks like we need to rebuild our relationship to the cultural needs of this country,’” I offer. “How about some research and experimentation to strengthen our place within the public imagination?”

“I don’t disagree,” Sokoloski nods, calmly. “But a good thing to come out of that hard mistake, is that it definitely enhanced an appetite for evidence within arts organizations. We received over 80 applications to take part in our data learning intensive following that exit strategy. Frédéric Julien [cultural policy leader with CAPACOA] was offering work on data years before us, and he kept warning me, ‘you’re going to have a hard time getting people engaged.’ Now, arts organizations are making this a priority because they recognize the need to generate and use their data better.”

Somewhat mollified, I offer my final question, “what’s your worst-case, best-case scenario here?”

“Doom and gloom?” Sokoloski squints at her ceiling. “I would worry that some framework gets taken up in rigid ways. That this gets operationalized as a standard of excellence to comply with, rather than scaffolding to build our own curiosity about ourselves.”

I recall Geoffrey Crossick warning us of something similar in his presentation to Toronto arts leaders years ago. The perennial risk of Goodhart’s Law — when a measure becomes a target, it ceases to be a good measure. That is, when we orient ourselves towards satisfying the measure of value, rather than the processes producing it, we disorient the very process we were hoping to strengthen. “And the optimism?” I ask.

“My optimism, and I see many reasons to be optimistic about this,” says Sokoloski, “is that there’s buy-in. That people get excited to use evaluation as a way of understanding the arts. That we are eager, proud, and capable of representing ourselves better — that regular citizens understand the different values the arts bring to our communities, and that our world changes as it starts to recognize the importance of those values.”

It’s a hefty vision, but a compelling and essential piece of the puzzle in securing the place of art in our worlds today.