“We matched participants to similar students”: what that buys, and what it can’t
For a long time, the sentence “participants were matched to similar non-participants on grades, attendance, and demographics” quietly ended my skepticism. It sounds like the problem of comparison has been handled. The groups look alike; surely the difference that remains is the program.
What changed my reading was one question, which I now ask about every matched study: if two students look identical in the data, why did only one of them sign up?
What matching actually does
Matching does one thing, and does it honestly: it makes the comparison group resemble the participants on the variables you matched. Same prior grades, same attendance band, same school poverty level. That is real progress; an unmatched comparison would be worse. After matching, the observable excuses are gone.
What it cannot do
The trouble is that the reason a student enrolls is usually not a column in the file. A tutoring program’s sign-ups differ from non-sign-ups in motivation, in whether a parent pushed, in whether a counselor happened to mention it. Matching on grades cannot touch any of that, because none of it is recorded. The technical phrase is “selection on unobservables”; the plain version is that matching removes the differences you can see, and the decision to participate is driven partly by differences you can’t.
There is a subtler version of the trap that took me longer to see. Match a signed-up student with low prior grades to a non-participant with the same low grades, and you may have paired the most motivated student in one group (low grades, still sought help) with an ordinary student in the other. Matching on the observable can quietly sharpen the unobservable difference, not shrink it.
The four things I look for in a matched study
A balance table, before and after. Matching is a claim about achieved similarity, and the claim is checkable: the same baseline-equivalence table any comparison-group study owes, shown for the matched sample. No table, no claim.
The enrollment story. I try to write one honest sentence about how a student ended up in the program. If the sentence contains a word like “chose,” “volunteered,” or “was referred,” I ask whether two students with identical files could plausibly differ on exactly that word. Usually they could.
Who got dropped. Matching discards participants with no good counterpart, which changes the question from “did the program work?” to “did it work for the kind of participant who has a look-alike?” A report should say how many were dropped and who they were.
A sensitivity analysis. The best matched studies ask: how strong would a hidden difference have to be to erase this effect? Reporting that number doesn’t remove the assumption; it prices it. When it’s missing, I price it myself, pessimistically.
The right amount of trust
None of this makes matching a trick. It is an honest tool for a narrower claim than the one usually attached to it, something like: among students who look alike on what we measured, participants did better. Studies that state the claim at that size, show the balance, and price the assumption are the ones I trust most. The reassuring sentence at the top was never the evidence. The table under it is.
I build baselinr, a small R package for exactly that balance table (WWC-aligned, before and after matching), and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.