Skip to main content

Personal Affect: The Impact of Measuring Impact

By Suz Curtis

From Gabriella Cowperthwaite's 2013 documentary 'Blackfish'. Courtesy of Magnolia Pictures.

Every film tells a story. Bottom line. But some films have a double bottom line, a two-fold mission: to tell a story, and to propel social change. To make an impact.

In the past decade, the means of measuring a film's social effect have grown increasingly sophisticated, inspiring numerous studies that measure impact—among them, Participant Media's 2014 study Storytelling Matters: Measuring the Social Impact of Entertainment on Audiences; the Media Impact Project, based at the USC Annenberg Norman Lear Center; and, from the Center for Media and Social Impact at American University, Assessing the Impact of Issues-Focused Documentaries: Research Methods and Future Considerations.

But what's all this measuring done for us lately? What's in it for the filmmaker? Three leading impact producers and strategists—Jessica Clark, Caty Borum Chattoo and Jennifer MacArthur—attempt to explain.

Clark directs the media strategy and production firm Dot Connector Studio, and serves as research director at Media Impact Funders, a network of representatives from the philanthropic community who work together on media and technology issues. She has spent more than a decade considering matters of impact. But don't talk to her about measuring it. "The first problem is people keep talking about it as 'measuring,'" she says. "It sounds frightening, or like you're in school getting graded."

Clark explains the value of a filmmaker understanding a work's impact: "If you're going to spend five or ten years of your life trying to tell a story, then you should have some sense of to whom you're telling the story and why. And whether they're actually listening, and what happens next."

Not to mention impact measurement being built into the user's platform. "There are a lot more metrics available to us now," she notes. "Filmmakers are trying to figure out how to operate in a multi-platform environment; there are lots more options for distribution, some of which come directly with calls to quantify views or engagement built into their digital structure."

Chattoo, who recently took over for Pat Aufderheide as director of the Center for Media and Social Impact, also avoids the "M-word." "I talk about studying the effects of storytelling," she maintains. "For me, studying means we can examine all the different ways documentaries can have an impact, and some of them are not quantitative. I think even the word 'measuring' connotes numbers. And a lot of it is numbers, but it's also a triangle of measuring knowledge, attitudes and behaviors."

MacArthur is a multi-platform communications, distribution and engagement strategist with a focus on social issue documentary. She offers a practical reason to embrace the topic. "Understanding the potential impact of your film and the ability to articulate it makes a big difference for filmmakers when they're trying to raise money for their film," she says. "The whole conversation around impact is driving a lot of the agenda in the funding community, particularly established foundations."

As for the measuring itself, Chattoo explains prevailing methodology: "We study the effects of the storytelling itself. You actually have people watch something; or after they watch something, you examine how their perspectives have changed, how their awareness has changed, and what they might do differently having watched it. Is the film out in the world having a kind of social impact that you can trace or talk about? If you have ten stories of people around the country who've started a recycling program because of a film they saw? That counts. It's not all big, quantitative data."

"There are two sets of circumstances to consider," Clark adds. "One is, what's their issue and where are they entering into the debate? The way they think of their outreach plan—or the way they structure the storytelling in order to target a particular outcome—has to do with how they think of themselves. Are they a storyteller, journalist or activist? Secondly, where is the issue in the public space? Is it a point where people don't know about it, and so there is awareness-building? Is it a point at which there's a clear pivot—a bill, an event or crisis point? Are they tracking people?

"Given all that, whom do they want to reach, and where?" Clark continues. "If you're targeting people under 20, it's a lot different from targeting people over 50. It takes research and design thinking. You have to go out into the field you're trying to affect and research it."

MacArthur offers that it's important to ask the right questions. "Look at the value of using art as a tool," she advises. "What's your strategy for using that art? How do you connect it with audiences? Did you execute on that in the best way possible? Those are the things I think can be evaluated or assessed, but not the film itself; the way you use the film. As a strategic thinker, do you understand how this as a piece of art can work in this moment?"

From Gael Garcia Bernal and Marc Silver's 'Who is Dayani Cristal?' Courtesy of Sundance Film Festival.

Or, as Chattoo explains, "The important questions in all of this are not, 'Should I measure impact and is there one way of studying it?' It's to first step back and say, 'What do we think this might do and what do we hope it will do?'" Using these questions as a rhetorical lens will guide the research and shape the study for maximum effect.

The impact studies are designed to be filmmaker tools, and MacArthur recommends one in particular: Who Is Dayani Cristal?, Gael Garcia Bernal and Marc Silver's 2013 film that addresses immigration. The study walks the reader through the impact team's position, methodology, strategy, success stories, tools and logistics. Such reports, MacArthur says, "help the filmmaker understand how to work deeply in communities with your film."

A filmmaker's intention does not always matter when qualifying a social change film. For example, Chattoo conducted an analysis of the 2013 documentary Blackfish. Her research showed that filmmaker Gabriela Cowperthwaite didn't set out to make a social change movie per se, but rather offered viewers a chance to decide for themselves. "She made the film about a character, which she should do as an artist," Chattoo maintains. "The story was so clear and the villain was so clear that the audience felt moved to do something about it. They did pre- and post-screening surveys, and they wanted to know if public opinion changed around this. And they found consistently that it did."

All three strategists also recommend a tool offered by BRITDOC: The Impact Field Guide and Toolkit. "It is specific to a particular type of impact strategy that is going to be broadly useful for getting filmmakers acquainted with basic concepts, but is geared toward mostly films that are advocacy-oriented or public awareness films, social issue marketing films," MacArthur says.

The abundance of tools, methodologies and data invites endless quantitative analysis. But there is an approach that considers a film's impact in a wider context. MacArthur sees this "bottom-up" versus "top-down" tension playing out in the field's strategic trends, which tends to be "geared toward elites," she says. "I mean that in the way of people in leadership positions and working at the policy level. But there are alternative approaches to this that are much more community-based, focused on keeping the power, and the way of making change is focused on what the community does—local, regional. The real focus on the work is on movement-building, education, engaging people in dialogue over time, cultural change and shifts. These are the things that are hard to measure. We are in a period of flux in terms of how funders understand these things."

After attending an Impact Distribution Lab presented by BRITDOC, Tribeca Institute, Sundance Institute, Ford Foundation and others, MacArthur noticed a trend in the conversation: "Two camps emerged: People saying, 'What about movement and community-building and the value of that and how it impacts change, versus the individual behaviors or actions of people?'

"We do have to professionalize as a field," she continues. "There's a way people talk about this—that the work is difficult and each piece of media is so different that it's hard to articulate sets of models that can be replicated that people can use. I think that's a bit of a cop-out. We have to get to a point as a field where the tensions we're seeing internally about ways to measure impact are really philosophical questions about what is impact and how people articulate change and power dynamics."

As technology, methodology and social movements and trends shape our understanding of impact, Clark explains that impact producers have a double bottom line of their own. "The point of the [impact producer] field is to make better films that move more people, and that's what we're trying to figure out. We're not saying filmmakers should move people in a particular direction. We just think that we should be paying attention to where."

And ultimately, the considering, studying and measuring of impact moves documentaries forward, according to Chatoo. "Studies around impact of documentaries should celebrate the art form," she maintains. "While other entertainment media has box office and attendance to measure, documentaries have this whole other story they get to tell, about how their work matters in the world."


Suz Curtis will graduate from UCLA's School of Theater, Film & Television in June 2015 with an MFA in Screenwriting. She's worked with Ladylike Films on the documentaries Somewhere Between, Code Black and PBS' Makers. Her adaptation of Death from Winesburg, Ohio will be distributed online as a short film, produced by James Franco with Rabbit Bandini Productions. She lives in Los Angeles.