The almost-experiment hiding in every cutoff
Education runs on cutoffs. Score below this line and you’re assigned to remediation; above that one and you win the scholarship; the class splits in two when enrollment crosses forty. Every one of those lines quietly mass-produces something precious: pairs of students who are nearly identical, one point apart, treated completely differently.
Drawn as a picture, outcome on one axis and a break at the cutoff line, it looks like a card trick, and my first instinct was to hunt for the catch. That is the whole idea of regression discontinuity, and it has been since two psychologists noticed it in 1960 while studying National Merit awards. Students at 79 and 81 points didn’t choose their side of the line the way program volunteers choose to enroll; the cutoff chose for them. Right at the line, assignment is close to a coin flip, and a jump in outcomes right there is about as believable as quasi-experimental evidence gets. Evidence clearinghouses agree: it is one of the few designs the What Works Clearinghouse will rate as highly as a randomized trial, when it is done well.
“When it is done well” is carrying weight in that sentence. Here is what I check, in order, when a report says “we used a regression discontinuity.”
Can anyone nudge themselves across the line? The design dies quietly if people can sort. If students can retake the test, if teachers round a 59 up to a 60, if families know the threshold in advance, then the two sides of the cutoff are no longer alike; the strivers have bunched up on the winning side. There is a standard check, plotting the density of scores around the cutoff to look for suspicious pile-ups, and a report that doesn’t show it is skipping the design’s load-bearing wall.
Does anything else change at the same line? Cutoffs are administratively convenient, so programs stack onto them. If the same score that assigns tutoring also triggers a parent letter and a schedule change, the jump you see is the bundle, not the tutoring.
Can I see the jump? The great virtue of this design is that the evidence is a picture: outcome on the y-axis, running score on the x-axis, a visible break at the line. When a paper reports a discontinuity but shows no scatter plot, or the plotted jump is invisible without the fitted lines to guide your eye, my confidence drops accordingly.
Does the effect survive the bandwidth? The analyst chooses how wide a window around the cutoff to use. Narrow windows compare the most similar students but leave few of them; wide ones buy sample size with comparability. An honest report shows the estimate across several windows. An effect that appears at exactly one bandwidth was found, not estimated.
Who is the effect about? This is the design’s built-in modesty, and it gets lost in summaries. A discontinuity estimates the effect for students near the cutoff. What remediation does for a student one point below the line may say little about what it does for a student thirty points below, who is the student the policy debate is usually about. The design answers a narrow question extremely well, and the fine print is the word “narrow.”
The catch I was hunting for turned out to be real, and it is the list above; the design is only as magic as its checks. None of them needs software; they need the report to have shown its work, which is the same standard as the rest of my reading checklist. A discontinuity study that shows the density check, the raw picture, and the bandwidth sweep has done something genuinely close to an experiment, without anyone having to randomize a child. One that shows a coefficient and a citation has shown me a line in a table.
I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.