Difference-in-differences and the leap of faith you can photograph
If you read education policy studies, you have met this design more than any other, whether or not the paper said its name. A state adopts a policy and its neighbors don’t. Compare the change in the adopting state to the change in the others, and the difference of those differences is the estimate. Two subtractions: the first removes everything permanently different about the state, the second removes everything that happened to everyone that year.
It is older than it looks. In the 1850s, John Snow noticed that one London water company had moved its intake upstream of the sewage while a competitor serving the same neighborhoods had not, and compared how cholera deaths changed for the two sets of customers. He didn’t need the two customer groups to be identical. He needed something subtler, and that subtler thing is still the entire game.
The assumption you can’t test
Difference-in-differences works only if, absent the policy, the two groups would have moved in parallel. Not been equal; moved in parallel. That counterfactual is unobservable by definition, which is worth sitting with for a second: the workhorse design of policy research rests on an assumption no dataset can verify. The first time this clicked for me, I thought I had misread it. A design this popular, resting on a leap of faith?
The redeeming feature is that the leap of faith leaves footprints. If the two groups were moving in parallel for the five years before the policy, believing they would have continued for one more year is a small step. If their gap was already widening, the design is dead on arrival, and no regression rescues it.
What I look for, in order
The picture. Outcome over time, both groups, adoption year marked. A flat pre-policy gap is the design showing its work, the same way a discontinuity study owes you the scatter plot. A paper that presents the two-by-two table of averages without the trend lines is asking you to take the photographable part on faith.
Why the policy happened. Policies are not raindrops; states adopt them in response to something. If a state passed a tutoring law because its scores were sliding, its trend was already different from its neighbors’, in exactly the way the design can’t absorb. The question “what prompted this policy, and was it moving the outcome already?” does more work than most robustness tables.
Who is in the comparison. “All other states” sounds neutral and rarely is. The honest version says why these comparisons and shows the result holds when you swap them, which is the same discipline as asking who got matched to whom.
Staggered adoption, handled or not. When many states adopt in different years, the naive version quietly compares late adopters to early ones already treated, and the average can bend or even flip. The fixes exist and are standard now; a recent paper that doesn’t mention the issue is dating itself.
None of this needs code to check, only the willingness to look for what’s missing. Parallel trends really is a leap of faith. The good studies photograph the runway before they jump, and hand you the photo. The rest describe the feeling of flight.
I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.