The students who disappeared are the finding
Take an evaluation with a healthy gain for the program group, and flip back to the sample table. One group shrank by a tenth and the other by a quarter before the outcome test, and the gain was computed on whoever remained. Nothing about the arithmetic is wrong. The question, which took me a long time to put into words, is what the arithmetic is still measuring.
A trial is only randomized on day one
Random assignment buys you two groups that are alike, once, at the start. Every departure after that is chosen, by a family that moved, a student who stopped coming, a school that quit the study, and chosen departures re-sort the groups. If the program’s strugglers give up on the study while the control group’s strugglers have nothing to give up on, the surviving program group is now selected for persistence in a way its comparison is not. This is the enrollment problem sneaking back in through the exit door, after randomization was supposed to have solved it.
That is why attrition is not a footnote about sample size. The evidence standards used to review education RCTs treat it as the first gate: the What Works Clearinghouse rates a randomized trial down if attrition is high, before anything else about the analysis is considered.
Two numbers, and the quieter one matters too
Attrition comes in two flavors, and reports love to disclose the first while staying vague on the second. Overall attrition is how many outcome measurements went missing in total. Differential attrition is the gap between the groups, and it is the louder alarm, because it means whatever drove students out operated differently on the two sides.
But equal rates are not automatic safety. Ten percent can leave each group for different reasons, the discouraged leaving one arm and the upwardly mobile leaving the other, and the comparison bends with no differential attrition at all. The rate is observable; the reasons mostly aren’t. Which is why the honest report does more than disclose percentages.
What the careful ones show
Three things, none of them exotic. A flow diagram: how many were assigned, served, measured, in each group, so the reader isn’t reconstructing it from footnotes. A leavers-versus-stayers comparison: the same baseline table the study already built, rerun on who dropped out, so you can see whether the missing tenth was the struggling tenth. And an analysis that keeps everyone assigned in the group they were assigned to, with a sensitivity check that asks how bad the missing outcomes would have to be to change the conclusion.
When a report instead analyzes “completers,” praises their gains, and moves on, question three of my reading checklist has been answered, just not in words. I say this with sympathy; a completers analysis is usually the easiest query to write, and it is easy to forget that it answers a different question.
The thirty-second version
Find the number randomized and the number analyzed, per group; the distance between them is where the story lives. Ask whether the leavers looked like the stayers at baseline. And check whether the headline number is about everyone who started or only those who finished, because “the program worked for the students who stayed” is sometimes the program’s success and sometimes its selection mechanism, and the report’s job is to help you tell which.
I build baselinr, a small R package for baseline-equivalence tables, including the leavers-versus-stayers kind, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.