Education Methods

Difference-in-differences and the leap of faith you can photograph

The workhorse design of education policy research rests on one untestable assumption: that two groups would have moved in parallel without the policy. You can’t prove it. You can, and should, demand the picture that makes it plausible.

Oct 8, 2026

If colleges were employers, two-thirds of those we can audit would fail the government’s own fairness test

Employment law flags a hiring process when one group’s success rate falls below 80% of the highest group’s. Apply the same arithmetic to college completion, and 66% of auditable institutions fail it for Black students.

Oct 1, 2026

The students who disappeared are the finding

A randomized trial is only randomized on day one. Every student who drops out afterward quietly re-sorts the groups, and the report’s most important table is the one comparing who left with who stayed.

Sep 24, 2026

The two hours that make an evaluation believable

When large medical trials were required to name their primary outcome in advance, the share reporting significant benefits fell from 57% to 8%. Pre-registration is the cheapest credibility a study can buy, and education finally has the infrastructure for it.

Sep 10, 2026

The almost-experiment hiding in every cutoff

Regression discontinuity is the most trusted design after the randomized trial, and the trust is earned only under conditions you can check from the report: a cutoff nobody can game, a jump you can see, and an effect that survives the bandwidth.

Aug 27, 2026

The evaluation came back null. Now what?

In 141 large education RCTs covering 1.2 million students, the average effect was 0.06 standard deviations. Null results are the normal case, not the scandal. Here is the order in which I think one through, and how to write it up without burying it.

Aug 13, 2026

“We matched participants to similar students”: what that buys, and what it can’t

Matching makes two groups comparable on the variables in the file. The reason someone enrolled is usually not in the file. How I read matched studies now, and the four things I look for.

Jul 30, 2026

How I read an evaluation report I’m inclined to believe

Ten questions I run through before trusting an education evaluation, in the order I ask them. None requires statistics beyond arithmetic; most can be answered from the report itself.

Jul 16, 2026

I ranked the same eight universities two defensible ways. The list nearly reversed.

A small experiment with real federal data: blend graduation rate and Pell share into one rating for Washington’s public universities, then move the weight and watch the order.

Jul 2, 2026

More controls ≠ less bias: the variables you should never put in your model

Adding a control and watching the coefficient move feels like being careful. Sometimes you just fed your model a fresh bias.

Measurement before models: the check most evaluations skip

A noisy outcome quietly shrinks the effect you can detect. Reliability and validity aren’t box checking; they decide whether a real program looks like it worked.

What a ‘teacher effectiveness’ score actually measures (and what it doesn’t)

Value added scores are noisier and more assumption laden than the tidy percentile suggests. They blend real teaching with sorting, test error, and plain luck.

Effect sizes for education: what WWC actually wants

Cohen’s d, Hedges’ g, and the small sample correction. Which standardized difference to report, and why getting it lazy quietly changes your conclusion.

Your baseline table is doing more work than your p value

Baseline equivalence carries the credibility of a whole impact evaluation, and most of us assemble it last, in a hurry. Here’s how the field actually wants it reported.

Before you compare two groups’ survey scores, check this

Comparing latent construct means across groups assumes the scale means the same thing to both. When it doesn’t, the difference is the instrument, not the construct.

Students aren’t independent: the clustering mistake that makes everything look significant

Treat clustered students as independent and your standard errors shrink, your p values glow, and your significant result is a mirage. The fix is routine.

Your text classifier is 85% accurate. That error doesn’t vanish when you run the regression.

Treating predicted labels as ground truth quietly biases everything downstream. With differential error it gets worse, and the accuracy number hides all of it.