Measuring Impact in Other Fields

If a history and literature review of how journalism has approached its influence and impact is useful as a look down deep into the topic, then this next section may be useful as a survey of adjacent fields’ approaches. Some pertinent professions’ practices are briefly outlined below.

Impact and metrics in development and health

Journalism researchers looking for expertise in achieving and evaluating impact can turn to the development fields, in particular the NGOs, foundations, and international organizations focused on health and social well-being. Their best practice, when planning an intervention, is to develop a “theory of change.”⁴⁷ The first step is to envision and articulate the ultimate outcome (e.g., bringing down rates of infant mortality in Afghanistan from 115 per thousand to forty per thousand), then work backwards through each precondition needed to change from the status quo to the desired condition. One of the many preconditions might be all children getting vaccinations, for which a precondition is parents knowing and agreeing to vaccinations, for which a precondition is an effective public information campaign, and so on. At each step the theory of change relies on cause and effect models that are accurate for the local environment. Ideally each intermediary step has indicators, which can be meaningfully measured.

Funders in the development field have also driven the adoption of standardized metrics to express the value of their outcomes. The DALY and QALY, favored by parts of the Bill & Melinda Gates Foundation and the World Health Organization, exemplify this practice, but the fundamental idea (having a comparable metric for expressing outcomes) underpins similar methods used by the Hewlett Foundation, the Acumen fund, and the Washington State Institute for Public Policy, among others.

In 1994 a movement to professionalize health development reached a milestone by establishing the DALY, or Disability Adjusted Life Year, intended to be a single metric to capture the impact upon a human of any condition, such as a disease or a trauma.⁴⁸ (Development organizations’ interventions aim to reduce that number.)

Using this framework, individuals in the target population experience a condition that burdens them a given amount. For example, total blindness is expressed as a sixty-percent burden, death is a 100-percent burden, whereas protein malnutrition to the point of wasting is considered a 5.3-percent burden. If a person lives with blindness for ten years, they would have lost six DALYs. Protein malnutrition for ten years becomes 0.53 DALYs, and so on. The figure can then be combined into a formula to express costs and benefits across populations: An intervention costing five dollars per person which prevents otherwise certain blindness, given to 100 five-year-olds with a life expectancy of seventy-five years, might be expressed as:

100 * (75-5) * 60% = 4,200 DALY

at a cost of

100 * $5 = $500

So this intervention has a cost per DALY of about 12c ($500/4,200) and thusly can be compared to any other intervention.

However, many readers will immediately see that while this might be precise, it has many points incorporating assumptions and estimations which risk losing accuracy and even meaning, while also flattening the lived experience of many individuals into a single number. Within the rich literature critiquing DALY, a Health Policy paper published one year after its introduction acknowledged limitations—including that the DALY sits within a narrow utilitarian value system—but did not wholly reject it.⁴⁹ Three years after DALY’s introduction, two more economists from Oxford and Harvard, Sudhir Anand and Kara Hanson, detailed both technical and ethical problems, concluding that DALY’s results would be practically flawed and decisions based on them would also be inequitable.⁵⁰ As late as 2014, Princeton researcher Rachel Parks charted the rise, resilience, and continuing use of the DALY through important centers of global health policy.⁵¹

Impact-minded journalists may be interested in the DALY for two main reasons, aside from its persistent power. Philanthropists have the option of giving their money to organizations that can express a likely value of their intervention in a simple number. (Responsible organizations will include caveats and acknowledge uncertainty, but at least they offer a comparable data point.) However, the underlying theory of change in journalism is far less direct than health organizations’. Journalists may need to articulate that their case for funding must embrace that uncertainty. For example, in James T. Hamilton’s book Democracy’s Detectives, he conducts an economic analysis of the cost of producing investigative reporting versus the monetary social value to provide a quantifiable economic societal cost savings.⁵²