Communicating Deference to Journalistic Judgment

One of the most popular and widespread suppositions about big data is that the existence of new sources of data and new ways to process them will obviate the need for expert intuition and judgment, which has, the narrative goes, repeatedly proven itself to be unreliable. In a typical example, Economist editor Kenneth Cukier and Oxford professor Viktor Mayer-Schönberger wrote, “the biggest impact of big data will be that data-driven decisions are poised to augment or overrule human judgment.”¹² In journalism’s more traditional corners, there is a fear that metrics will do more overruling than augmenting. One of the challenges for Chartbeat staff, then, is to present itself and the company’s products in a way that will assuage these worries and earn journalists’ trust. The company does this in two ways. First, staff members rhetorically defer to editors’ judgment—both in verbal interactions with clients and within marketing materials. In a client meeting I observed, a Chartbeat employee suggested the client look at data about its most valuable traffic source and added, “you already know it intrinsically, but the data confirms it.” (Of course, Chartbeat also has to avoid going too far with this type of message, lest the dashboard be considered unnecessary.)

The second way in which Chartbeat works to communicate deference and build trust is with the metrics themselves. The new Chartbeat Publishing dashboard prominently features two metrics in the top left-hand corner: engaged time, which is an average measure of how long people spend engaged with a site’s content, and recirculation, which is the percentage of people who visit at least one additional page on a site after the one at which they arrive. Just below these figures is a section on visitor frequency, which divides readers into three categories—new, returning, and loyal—based on how often they visit a site. In explaining the impetus for these particular metrics, Chartbeat employees and marketing materials position them against page views and other metrics that have a reputation for incentivizing the production of clickbait. By contrast, the company argues, Chartbeat’s metrics are designed to reward “high-quality content,” meaning the kinds of rigorous, thoughtful reporting and writing that are central to journalists’ professional identity. In a meeting I observed in which the Chartbeat team prepared for the product launch, the alignment between Chartbeat Publishing’s metrics and traditional journalistic values was a central selling point. As one employee put it, “this is the core of what we’re trying to do—reframe an audience from chasing meaningless metrics and starting out fresh every morning, to building a loyal and returning audience that [the client] can monetize in a variety of ways.”

Just as noteworthy as which features Chartbeat Publishing includes are the ones it leaves out. The dashboard doesn’t make recommendations about what kind of content to produce or where to place content on a page. This omission is not due to technological limitations or because such a feature would be ineffective at growing clients’ traffic; rather, it is a conscious attempt to avoid alienating journalists. When asked about Chartbeat’s strategy, a member of the development team explained:

We had a competitor who made a tool that made suggestions to editors …And it was like, “put this here, promote this story.” And editors were like, “I’m not using this damn thing …You’re telling me to put stuff in the lead spot I would never put there.” So we said, “listen, we’re not taking away your job; we’re enhancing your ability to make those decisions.”

Indeed, an anecdote about a Chartbeat competitor that exemplified this dynamic was invoked so frequently in internal meetings and interviews that I began to consider it part of the company folklore. At the Guardian Changing Media Summit in 2013, the digital editor of the British tabloid The Sun served on a panel with the founder and CEO of Visual Revenue, a predictive analytics company that provides algorithmic recommendations to news organizations about story placement and assignment. In speaking about his experiences using the product, Derek Brown, The Sun editor, said:

It’s a really valuable tool, but the one thing I’ve always said to Dennis [founder and CEO of Visual Revenue] …is that you kind of have to ignore it sometimes. Great example, last week: New Pope [was elected, and] Visual Revenue is telling us, screaming at us, “nobody is interested in this story! They’re far more interested in the Katy Perry story. Put that above the Pope. Put the Pope way down at the bottom of your home page. Nobody really wants to read it on your website.” And of course, at that point—and there’s lots of examples of that every day of the week—the human being has to intervene and say, “okay, they may be more interested in Katy Perry in a bikini today. However, the Pope is a far more important story.”¹³ That an editor at The Sun, a tabloid newspaper, so frequently found Visual Revenue’s recommendations inappropriate is indicative of the fraught relationship that news organizations of all stripes can have with analytics companies. At Chartbeat, “Katy Perry versus the Pope” (as the story was sometimes referred to during meetings) was a powerful symbol. This was in part because, as one staff member put it, “we don’t feel that’s how data should be used.” But it also served as a cautionary tale of the kind of overstepping that could damage an analytics company’s credibility with news organizations. The story became a vivid reminder of the delicate balance Chartbeat had to strike with clients. The company needs to show that data has something to contribute to decision-making without stepping on editorial toes.