The evaluation came back null. Now what?

In 141 large education RCTs covering 1.2 million students, the average effect was 0.06 standard deviations. Null results are the normal case, not the scandal. Here is the order in which I think one through, and how to write it up without burying it.

Here is the most useful fact I know about education evaluations. In 2019, two researchers collected 141 large randomized trials commissioned by the two most serious funders of education evidence in the US and the UK, covering 1.2 million students. The average effect across all of them was 0.06 standard deviations, roughly nothing, and the confidence intervals were so wide (0.30 standard deviations on average) that many trials couldn’t distinguish “small effect” from “no effect” at all.

So when your evaluation comes back null, the first thing to know is that you are holding the normal result. I find this fact genuinely comforting, the way weather statistics are comforting when your picnic gets rained out in Seattle. The question is not how to apologize for it. The question is what kind of null you have, and I work through that in a fixed order.

First: is the null informative, or just imprecise?

“No detectable effect” covers two different findings. A tight confidence interval around zero says something real: effects big enough to care about probably aren’t there. A wide interval mostly says the study couldn’t answer the question, which is what happened in many of those 141 trials. The way to tell them apart is to translate the interval into units a reader can feel: “we can rule out gains larger than a month of learning” and “we can’t rule out anything between harm and half a year of gains” deserve opposite reactions.

Second: could the study have hidden a real effect?

Before concluding the program failed, I check whether the design could have produced a null even if the program worked.

• The outcome couldn’t see the change. An unreliable or off-target measure shrinks every effect toward zero, which is why I check measurement before models, and why an outcome that was itself machine-classified with error attenuates further.
• The score doesn’t mean the same thing in both groups. A null gap on a scale that functions differently across groups is not evidence of equality, in either direction.
• The groups weren’t comparable to start. If baseline equivalence failed, a real gain in one group can be canceled by a head start in the other. The same worry applies to matched comparisons, where the invisible reasons people enrolled can mask or mimic effects.
• The outcome was aggregated away. Composites hide movement; what a summary score actually captures and how the weights were set both decide whether a real change surfaces.

Third: is the null the honest number, at last?

The mirror image matters just as much. Analyses that treat clustered students as independent produce stars out of noise, and piling on control variables can manufacture effects as easily as destroy them. Sometimes a well-analyzed null is not the study failing; it is the first analysis in the room that stopped flattering the program.

Then write it like a result, because it is one

Lead with the finding, in the first paragraph, in plain words. State what the interval rules out, not just the p-value. Point to what success was defined as in advance, if anything was, since a null against a pre-named target is a result while a null against nothing is an anecdote. Keep subgroup patterns, labeled as exploratory, for the next study to test. And separate the two sentences a decision-maker actually needs: “it probably doesn’t work” and “we couldn’t tell,” one of which should appear, alone, in your summary. If you want to see these habits from the reader’s side, they are questions seven and ten on my checklist.

I’m still in grad school; nobody has yet paid me to deliver bad news to a funder, and I don’t imagine the room feels like a statistics seminar. But a clean null saves someone real money, usually the district about to scale the program. And it compounds: people who watch you report a null clearly will believe your positive finding next year at face value. In a field where the average large trial finds 0.06 standard deviations, the evaluators who can say “nothing happened” gracefully are not the failures. They are the ones telling the truth at the base rate.

I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.