Before going any farther, we should clarify what we mean by impact. To start, our base assumption is that journalism can and does have an effect on the world and it does so without necessarily becoming advocacy.i Unlike other work on this topic, we don’t offer a strict definition of impact. Based on our research, successful impact measurement can only happen if an organization has identified its institutional goals. From there, it can begin to measure elements that bring it closer to those goals (e.g., encouraging subject-matter influencers to discuss one’s work if the goal is to improve the credibility of one’s reporting). As a result, we created a loose impact framework as opposed to a strict definition.
This approach offered us two advantages. The first is that it exposed newsrooms’ current thinking about what they consider to be important events, allowing us to see the existing differences across organizations. The second is that it served as scaffolding for newsrooms that do not yet have an articulated understanding of their own goals. We found this was the case most often with national and international publications and less so with local, regional, or topic-based ones.
Maintaining standard terminology and vocabulary is a worthwhile goal, however. It goes without saying that the more newsrooms use common terms and tools, the more opportunity exists for contextualizing and understanding a project’s success. In Chapter 5 of this report, Recommendations and Open Questions, we discuss those factors that could make standardization possible in the future.
Our view of impact necessarily incorporates both qualitative and quantitative information. Many newsrooms interested in measuring qualitative events expressed their frustration at the limits of page view-driven decisions. They worried that only high-traffic stories get lauded and, consequently, shape the editorial agenda. In fact, when we started this project one of our main goals was to build a tool to better highlight qualitative events. If traffic is the only measure of success, then how can you show the value of the niche story on an important topic?
Through the course of our research, we also heard examples emphasizing the value of tracking existing and new quantitative measurements. One such story that stuck with us came from a mid-size, nonprofit newsroom producing a mix of investigative, political, and culture reporting. It explained how routine traffic to its stories is orders-of-magnitude higher today as compared to two years ago. The newsroom uses this information to show readers and funders that its organization is trending upwards. Taking this data in the aggregate, and not letting any one number guide strategy, the company is able to construct a narrative about its editorial reach backed by numbers, not discrete and varying qualitative events.
Jonathan Stray wrote about the difficulty of qualitative measurement: “Some events are just too rare to provide reliable comparisons—how many times last month did your newsroom get a corrupt official fired?”5Numbers do fill a valuable role in understanding organizational health, and we think removing them from the equation eliminates a potentially valuable lens through which to gauge success.
When we say “the impact of journalism” or “the impact of a newsroom’s work” we should also clarify the limits to what we can reliably study—that is to say, where we chose to start research for this project. From cultural commentary to the court reporter on a beat, journalism exists in so many varieties that it can make the question of journalism’s impact seem too large to tackle.
To narrow the scope of our research, our initial target newsroom was the small, nonprofit investigative organization.
Two elements informed this choice. We were most interested in journalism that seeks to address something about the world (this is often investigative work) and, therefore, allows newsrooms to more easily state tangible goals for their projects. For instance, did this illegal practice end? Did the government increase oversight? Are companies now following the law?
The second reason was organizational. Such newsrooms often look to grants or benefactors for funding; and these outside groups often require reports outlining how the organizations have used their money—hopefully guaranteeing that it was well spent. As a result, impact measurement is not a foreign concept to these benefactors, albeit still not an easy one.
We want to stress that these are neither the only nor necessarily the “best” examples of journalism’s impact. For example, looking at how media coverage can shape discourse is another fascinating and worthwhile area of study. We do, of course, remain cognizant of other forms of analysis, such as pre- and post-intervention surveys, which might fold into the NewsLynx platform in the future as more newsrooms adopt increasingly sophisticated techniques of impact measurement. Our focus here, however, is on the current needs and practices of investigative newsrooms.