The Problem of What to Count
Quantification always involves complex choices, even in the hard sciences. Although friction is a basic force of classical physics, it comes from micro-interactions between surfaces that aren’t fully understood. A high school physics textbook will tell you that we usually describe it with two numbers: the coefficient of static friction which is how hard you have to push to start sliding, and the coefficient of kinetic friction which is how hard you have to push to keep sliding. But more sophisticated measurements show that friction is actually quite a complex force. It also depends on velocity, and even on how fast you were sliding previously.15 Anyone working with friction has to choose how to quantify it.
Race is even more difficult to quantify, as are a great many things of social interest. it’s terribly easy to forget this complexity when you are looking at neat rows and columns of data.
A few years ago I worked on a story about gun violence. At the time there was a lot of popular discussion about “mass shooting” incidents, and whether they were or weren’t on the rise. But what’s a “mass shooting”? It seems like a single murder doesn’t count, so how many people must be killed at once before it’s “mass”? You have to answer this question before you can answer the question of whether such incidents are more or less common than before. I eventually chose four people as the minimum threshold for a mass shooting, because that’s what the data I had used. The creators of that data chose four because this is how the FBI counts “mass murders,” even though those aren’t quite the same thing as “mass shootings.” Responding to the interest in these events, the FBI later released its own data set of “active shooter” incidents, which it defined as “individuals actively engaged in killing or attempting to kill people in populated areas (excluding shootings related to gang or drug violence).”v
This is all somewhat arbitrary, and there is no “right” answer here. What you should count depends on what you care about, that is, it depends on the story you are attempting to tell. And after looking at the data you may realize that you want to count something else. Your initial story may turn out to be uninteresting, unfair, or just plain wrong.
It gets even trickier. Imagine tracking the prevalence of mental health issues such as “depression” or “borderline personality disorder,” which are short names for evolving ideas about diseases. The complex diagnostic criteria for these conditions, which used to be printed in thick handbooks, define a quantification process. Or think of the police officer who must record if a particular incident is “sexual harassment” or not. it’s easy to imagine that not every officer will have the same idea of what sexual harassment means. This can make the data maddeningly hard to interpret, not to mention unfair. Small differences in counting technique can and do become the focus of intense arguments.
Still we find some way to count. A quantification process formalizes the act of counting or measuring or categorizing and attempts to apply it consistently across many situations. That’s the whole point of standard units like meters and kilograms. But alas, many vital things do not have standard measures. How do we quantify more abstract concepts such as “educational attainment” or “quality of life” or “intelligence”?
In practice we end up replacing such rich concepts with much simpler proxies. We get “test scores” instead of “educational attainment” and “income” as a proxy for “quality of life,” while “intelligence” is today measured by a battery of tests which assess many different cognitive skills. In experimental science this is called operationalizing a variable, a fancy name for picking a definition that’s both analytically useful and practical enough to create data.
If you want to ask a question that only quantitative methods can answer, you have little choice but to make this switch from rich conception to repeatable measurement. But quantification can also force you to be clear. Trying to quantify might lead you to discover that you’ve been using certain words for a long time without really understanding what they mean—do you really know what “intelligence” is? Eventually a quantification of a thing can become the definition, as the IQ test did. This might be a clarifying improvement, or a narrowing of perception, or both. In any case, it is a choice that should be made consciously.
Usually there is some end goal, some purpose to collecting data, and you can ask whether any particular quantification method serves that purpose. And you can ask about the end purpose, too, the frame of the entire thing. Different quantification methods serve different stories.