Impact and Metrics in Information and Communication Worlds

Science and social sciences

Moving from the fields of development closer to the world of journalism, we pass academia and scientific research where there has been an established and highly structured metric for impact. When researchers publish formal articles in peer-reviewed journals, those publications will have an “impact factor,” calculated by taking the number of times the journal has been cited in a year and dividing by the total number of articles published over the previous two years, as recorded by Thomson Reuters’ “Journal Citation Report” (JCR). Tenure and promotion committees pay heed when researchers, especially those in the sciences and social sciences, publish papers in highly rated journals such as Science and Nature. However, the “JCR” does have its critics who say that Thomson Reuters’ database skews toward North America, its data is hard to verify, that the calculation cannot fathom whether the citation is approving, disapproving, or central to the citing author’s argument, and a host of other problems.

As a counterpoint to the “JCR” impact factor, some academic researchers have started to pay attention to “Alt-Metrics,” a system which also includes indicators of usage (in the form of views and downloads), peer review, discussion on social media, data usage, as well as citations.⁵³ Researchers add their own expert understanding to either of those frameworks of whether peers’ findings have become useful and built-upon in the field. For particularly important findings, studies are replicated; theoretically, when the research is done again, in the same way, it should produce the same results.

Advertising/marketing

Leaving aside, for a moment, the vast chasms of motivation, methods, and form (at least traditionally) that separate advertising and journalism, there are still similarities. Advertising intends to persuade the audience, as does some journalism. Advertising’s success metrics, however, are much more clearly linked to driving revenue for businesses (although that relationship became complicated in later years).

Indeed, the early performance indicators for marketing executives in products and services firms were found in the financial statements. In the 1950s and 1960s, according to marketing historian Bruce Clarke, most advertising was judged on whether revenue and/or operating profit went up (more sales being made or products commanding a higher price compared to the cost of production), and/or market share increased.⁵⁴ As advertising management became more sophisticated, executives argued that those indicators lagged their output by too much. Their solution was to measure softer, but standardized indicators: primarily, customer satisfaction, customer loyalty, and brand equity.⁵⁵ (At this point, journalists might start to see potential lessons: These labels refer to attitudes held by people exposed to media works, although the difficulty remains of proving causality in an intrinsically chaotic environment.) Customer satisfaction was expected to drive future sales revenue in volume and the higher prices customers would be prepared to pay because they were already satisfied with the product. The financial premium derived from satisfaction, loyalty, and audiences’ overall improved perception of the brand, in the form of increased revenue and lower subsequent customer acquisition costs, was called brand equity. Surveys of marketing professionals have revealed more attitudinal metrics, and myriad combinations of financial and non-financial measures, but they still follow the logic that a population’s perception of a product or service will influence the financial statements over time.⁵⁶ Marketing and advertising firms traditionally used surveys and focus groups to measure these attitudinal factors, although the advent of social media has produced a set of tools promising to deliver customer insights and measure the impact of marketing and communications.⁵⁷

Although journalists may balk at adopting practices from the marketing industries, that field has developed expertise in measuring the effects of their activity.

Documentary filmmaking

Journalists and documentary filmmakers are perhaps the closest relatives in this survey. The fields overlap. However, the inclusion of philanthropic funding sources in documentary economics precedes the rise of the online news organizations like ProPublica and The Marshall Project, which are strongly associated with impact and foundation-funded journalism. As such, the documentary community has established a practice and rhetoric around impact which is more recognized than in other mission-driven journalism.

Media-funding philanthropists, including the Ford Foundation, the Knight Foundation, the Bertha Foundation and the Sundance Institute, underwrote a guide to impact for the documentary field. Its roots lie in a model published by political and communication theorist Harold D. Lasswell in 1948. Lasswell said, simply, the best way to describe an act of communication was:

Who

Says What

In Which Channel

To Whom

With What Effect?⁵⁸

Jana Diesner, a researcher in computer science with relevance to impact assessment, applied this approach in a framework for measuring the impact of documentaries. She called it the CoMTI model, suggesting research techniques and suitable metrics for measuring results throughout four dimensions of a film: its content, medium, target, and impact. Within the impact dimension Diesner uses the preexisting idea that effects can be on individual, communal, societal, or global levels, which may take the form of changes to awareness, sentiment, and actions.⁵⁹ Although Diesner’s CoMTI paper was an intermediary step toward building a computational tool to analyze text from the network of stakeholders around a documentary, the model and its constituent parts is valuable as an extensive catalogue of metrics and research techniques for assessing journalistic impact.

Conclusion

The history of thinking around how journalism affects society supports a conclusion that journalists must accept that their work can and frequently does have consequence. However, unlike some of the other professional fields outlined above, it is easier to see something like influence—one element within a suite of factors leading to an end result, rather than the direct impact of a health intervention like immunizations or cataract surgery. Nonetheless, each of the impact and influence theories discussed above have their limitations. Journalism, which necessarily operates in the chaos of the uncontrolled real world, must expect to tackle uncertainty, which is antithetical to a stable and predictable theory of change.