How I read an evaluation report I’m inclined to believe
The evaluation reports that fool me are never the sloppy ones. They are the ones whose conclusion I already agree with. I’m a master’s student, so I read a lot of these on assignment, and the routine below is what I have settled on: ten questions in a set order, precisely so that my sympathy for the finding doesn’t get a vote. None of them needs software. Most can be answered from the report itself, and when one can’t, that absence is usually the answer.
1. Compared to whom? Before any number, I look for the comparison group. “Participants improved” is not a finding; students improve on their own, programs or not. If there is no comparison group at all, I stop expecting a causal claim and read the rest as description.
2. Were the groups similar before the program? The whole causal claim rests here, and it is checkable from one table. I wrote about what that table has to show and why a p-value is the wrong check; a report that never shows baseline equivalence is asking me to take the hardest part on faith.
3. Who disappeared? Enrollment counts at the start, outcome counts at the end. If a quarter of the treatment group is missing from the final table, the program’s effect is now entangled with who stuck around, and I want the report to say how the leavers differed, not just how many left.
4. Was the outcome measured the same way for both groups? Same test, same timing, same raters. A program grades its own homework more often than you’d think, sometimes literally. When groups are compared on a score, I also want to know the score means the same thing in both.
5. How big is the effect, in units I can feel? “Significant” tells me about sample size as much as anything. I look for the effect in months of learning, points on a real test, or a standardized effect size I can compare, the way evidence clearinghouses ask for it.
6. Were students counted as if they were independent? Twenty-five students in one classroom share a teacher, a curriculum, a snow day. Analyses that ignore this make everything look significant. The fix is standard; its absence in 2026 is informative.
7. How many outcomes were measured, and how many reported? Ten outcomes and one star is a different story from one pre-named outcome and one star. I look for a protocol, a pre-registration, or at least a full outcome table with the nulls left in.
8. What was controlled for, and why? More controls is not automatically better; some variables make the estimate worse when you adjust for them. I read the control list asking “is this a cause of enrollment, or a consequence of the program?”
9. Who paid, and who analyzed? Not as an accusation, as a base rate. Evaluations commissioned by the program’s vendor come out favorable more often, and the mechanism is rarely fraud; it is which outcomes, sites, and cohorts made it into the report. Independence is a design feature like any other.
10. What would failure have looked like? My favorite question, and the one reports answer least often. If every plausible result could be narrated as success, the evaluation was not a test. When a report states in advance what result would have counted against the program, everything else in it gets more believable.
What a checklist can and can’t do
Ten questions can’t convict a report, and passing them doesn’t make a finding true. Most of them cover mistakes that are easy to make and easy to miss when reading. This is triage: it tells me where to spend skepticism, which is always in shorter supply than I’d like, and it protects me a little from my own preferences. The reports I end up trusting most are rarely the ones with the biggest effects. They are the ones that make questions 1 through 10 easy to answer, because the authors asked themselves first.
I build baselinr, a small R package for the baseline-equivalence table in question 2, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.