There are so many policy questions that need answers during the COVID-19 pandemic. We would like to know how every intervention has affected the spread: have extended testing, school closures, stay-at-home orders, or even just officially-recommended social distancing made a difference? And how much?
Let me start with some bad news: we can never know exactly.
The reason is almost too simple: to truly know the effects of a policy, we need to watch the world unfold with the policy, then rewind, and watch the world unfold in its absence without changing anything else. If we could just compare the real world to this world that might have been – which social scientists call the “counterfactual” and we geeks call an “alternative timeline” – then we would know what effect the policy had. Like a control group in an experiment, the counterfactual anchors our measurement of a policy’s effects. But without a time machine, it’s unknowable.
I wonder if this is why so many people throw up their hands and say, “We just can’t really know anything about all this.” Or, perhaps more commonly, people latch on to evidence corresponding to their own inclinations, and it then seems obvious that their preferred narrative is true. As studies accumulate by the day, how is the general public supposed to decide which analyses are treating the data with respect and care, and which are shoving it into a pre-fabricated story mold? How can we judge?
This is where the good news comes in. If the problem is an unknowable counterfactual – and it always is! – then a study can provide solid, trustworthy results if it has developed a credible substitute. If we want to know the effects of stay-at-home orders on COVID-19 case growth, the challenge is to use all of the available information to estimate what case growth would have been in the absence of stay-at-home orders. This is far from impossible. Social scientists have been working on methods for developing credible counterfactuals for decades, and there are objectively better and worse ways to do it.
Consider an example. Suppose I’d like to know the effect of food banks on hunger. The data tell me that people living near food banks are hungrier, while people without a local food pantry are doing fine. More access corresponds to more hunger. In other words, food banks make things worse. Does this sound wrong to you? It should. It is not the data, but the analysis, that is creating the problem. It assumes that if neighbourhoods with food banks didn’t have them, their hunger levels would be more like neighbourhoods that currently don’t have food banks.
Yet, neighbourhoods without food banks are likely wealthier, with many advantages that improve their food access. They are not good counterfactuals because of important, underlying differences. There may also be a cause-and-effect relationship running in the opposite direction from what we hope to measure: if pantry locations were actually chosen to be in poor neighbourhoods, local hunger (in a sense) “causes” food pantry access. In the presence of both important secondary factors and these reverse causal paths, it is hard to identify the effect of a food pantry on hunger.