The Art and Science of Data-Driven Journalism

Data and Reporting Paired with Narrative

“The Prescribers,” ProPublica’s series on fraud and influence in the Medicare drug system, is a masterful use of data in investigative reporting.121connection between data-driven investigative journalism and government or corporate accountability. It’s far from the first at that outlet. “The project I’m most proud of is something I did before SOPA Opera, which was our ”Dollars for Docs’ project in 2010,” said Dan Nguyen, then a developer at ProPublica. “It started off with just a blog post I wrote to teach other journalists how Web scraping was useful.122to disclose what it paid doctors to do promotional and consulting work. My colleagues noticed and said that we could do that for every company that had been disclosing payments. Because each company disclosed these payments in a variety of formats, including Flash containers and PDFs, few people had tried to analyze these disclosures in bulk, to see nationwide trends in these financial relationships.”Nguyen explained that the ProPublica team wrote dozens of data scrapers to cross-reference their database of payments with state medical board and medical school listings. “For the initial story, we teamed up with five other newsrooms, including NPR and the Boston Globe, which required programmatically creating a system in which we could coordinate data and research,” he said. “With all the data we had, and the number of reporters and editors working on this outside of our walls, this wasn’t a project that would’ve succeeded by just sending Excel files back and forth.”123data-driven journalism can have an impact on patients, providers, private companies and universities; in short, an entire health care industry.“The website we built from that data is our most visited project yet, as millions of people used it to look up their doctors,” said Nguyen. “Afterwards, we shared our data with any news outlet that asked, so hundreds of independently reported stories came from our data. Among the results were that the drug companies and the med schools revisited their screening and conflict of interest policies.”