How I read a What Works Clearinghouse report (and what it can’t tell me)
The first time I used the What Works Clearinghouse to check an intervention, I treated the rating like a verdict: positive means buy, nothing means skip. I read the same pages differently now, because I finally understood what question the WWC is actually answering. It is narrower than the name suggests, and knowing exactly how narrow is what makes the reports useful.
The WWC reviews studies of an intervention and rates the evidence, not the intervention. Its reviewers ask: were these studies designed so that the measured effect is believably the program’s doing? That is an internal-validity question, the same one this blog keeps circling: randomization held together by low attrition, groups equivalent at baseline, outcomes measured properly. A strong rating means “this evidence would survive a skeptic.” It does not mean “this will work in your district.”
The order I read in now
Start at the bottom: how many studies? This is the number people skip, and it reframes everything else. An intervention can carry a positive rating on the strength of one qualifying study, sometimes with a few hundred students. That is not a scandal; a single good study beats ten bad ones. But “positive, based on one study in one city” and “positive, based on twelve studies across three grade bands” are different bets, and the rating alone won’t tell you which you’re holding.
Then the size, translated. The WWC reports effects in an “improvement index”: where the average treated student would land in the comparison group’s distribution, expressed in percentile points. Plus four means the median kid moves to the 54th percentile. It is an effect size wearing friendlier clothes, and the friendly clothes make it easy to forget that plus four is modest, and that plus four on average contains classrooms where nothing moved.
Then who was studied. Grade levels, student populations, settings. The report lists them plainly, and the list is where “works” quietly becomes “worked, for someone.” An intervention validated on ninth-grade English learners in urban districts is a hypothesis, not a finding, about your rural fourth grade.
Only then the rating. By this point it mostly confirms what the study count and design details already told you.
What the label can’t carry
Three things, honestly. Most interventions you look up will have no studies meeting standards, and absence of evidence is not evidence of absence; it usually means nobody has funded a rigorous test yet. The reports say little about cost, training burden, or what implementation actually took, which is where programs live or die in practice. And a review can only be as current as the literature; a rating from studies a decade old describes a program as it was then.
None of this is a complaint. A nutrition label doesn’t cook dinner either. The WWC does one hard thing well, filtering evidence through the same checks I’d want to run myself, and reading it with the grain of that one thing, instead of as a verdict, is what it took me embarrassingly long to learn.
I build baselinr, a small R package for WWC-aligned baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.