The order I read in now

Start at the bottom: how many studies? This is the number people skip, and it reframes everything else. An intervention can carry a positive rating on the strength of one qualifying study, sometimes with a few hundred students. That is not a scandal; a single good study beats ten bad ones. But “positive, based on one study in one city” and “positive, based on twelve studies across three grade bands” are different bets, and the rating alone won’t tell you which you’re holding.

Then the size, translated. The WWC reports effects in an “improvement index”: where the average treated student would land in the comparison group’s distribution, expressed in percentile points. Plus four means the median kid moves to the 54th percentile. It is an effect size wearing friendlier clothes, and the friendly clothes make it easy to forget that plus four is modest, and that plus four on average contains classrooms where nothing moved.

Then who was studied. Grade levels, student populations, settings. The report lists them plainly, and the list is where “works” quietly becomes “worked, for someone.” An intervention validated on ninth-grade English learners in urban districts is a hypothesis, not a finding, about your rural fourth grade.

Only then the rating. By this point it mostly confirms what the study count and design details already told you.

What the label can’t carry

Three things, honestly. Most interventions you look up will have no studies meeting standards, and absence of evidence is not evidence of absence; it usually means nobody has funded a rigorous test yet. The reports say little about cost, training burden, or what implementation actually took, which is where programs live or die in practice. And a review can only be as current as the literature; a rating from studies a decade old describes a program as it was then.

None of this is a complaint. A nutrition label doesn’t cook dinner either. The WWC does one hard thing well, filtering evidence through the same checks I’d want to run myself, and reading it with the grain of that one thing, instead of as a verdict, is what it took me embarrassingly long to learn.

I build baselinr, a small R package for WWC-aligned baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

The assumption you can’t test

Difference-in-differences works only if, absent the policy, the two groups would have moved in parallel. Not been equal; moved in parallel. That counterfactual is unobservable by definition, which is worth sitting with for a second: the workhorse design of policy research rests on an assumption no dataset can verify. The first time this clicked for me, I thought I had misread it. A design this popular, resting on a leap of faith?

The redeeming feature is that the leap of faith leaves footprints. If the two groups were moving in parallel for the five years before the policy, believing they would have continued for one more year is a small step. If their gap was already widening, the design is dead on arrival, and no regression rescues it.

What I look for, in order

The picture. Outcome over time, both groups, adoption year marked. A flat pre-policy gap is the design showing its work, the same way a discontinuity study owes you the scatter plot. A paper that presents the two-by-two table of averages without the trend lines is asking you to take the photographable part on faith.

Why the policy happened. Policies are not raindrops; states adopt them in response to something. If a state passed a tutoring law because its scores were sliding, its trend was already different from its neighbors’, in exactly the way the design can’t absorb. The question “what prompted this policy, and was it moving the outcome already?” does more work than most robustness tables.

Who is in the comparison. “All other states” sounds neutral and rarely is. The honest version says why these comparisons and shows the result holds when you swap them, which is the same discipline as asking who got matched to whom.

Staggered adoption, handled or not. When many states adopt in different years, the naive version quietly compares late adopters to early ones already treated, and the average can bend or even flip. The fixes exist and are standard now; a recent paper that doesn’t mention the issue is dating itself.

None of this needs code to check, only the willingness to look for what’s missing. Parallel trends really is a leap of faith. The good studies photograph the runway before they jump, and hand you the photo. The rest describe the feeling of flight.

I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

What the flag does and does not mean

I want to be careful about what this number means, because the temptation to overread it is strong. An institution below the line has not been shown to discriminate. Completion gaps carry everything students bring with them, from prior schooling to money to the hours they can give to coursework. The four-fifths rule was built for hiring decisions, and lifting a threshold out of employment law and dropping it onto a multi-year educational process is the kind of move fairness researchers call a portability trap (Selbst and colleagues, 2019).

What the rule offers is different: a common, transparent trigger. In hiring, a number like this obliges someone to look. In higher education it obliges no one to do anything. That asymmetry is what stayed with me while I was writing the paper.

What the distribution says

The distribution tells me more than the headline share does. Most institutions are not sitting just under the line: the median is 0.70, a fifth of auditable institutions are below 0.50, and fewer than 7% complete Black students at or above their White rate. The result also survives the checks I could throw at it, including a stricter reference group, an absolute-gap measure instead of a ratio, and different cohort minimums. Some of the concentration in low-completion institutions turns out to be partly a ratio artifact, and the paper says so. The national pattern does not go away.

Why borrow a hiring rule at all

Partly because higher education has no agreed threshold of its own. Completion gaps are reported in many dashboards and cut many ways, and none of them comes with a line that says this one warrants a closer look. Employment law adopted a crude but usable line fifty years ago. Borrowing it does not settle what a college owes its students. What it gave me was a single number that shows how this landscape would look if the standard we already apply to hiring applied here too.

The audit is descriptive, not causal, and the data are public: IPEDS graduation rates, 2018 through 2022, via the Urban Institute Education Data Portal.

The full paper: Liang, Y. (2026). A national disparate-impact audit of college completion equity. Educational Policy. Advance online publication. https://doi.org/10.1177/08959048261489777

Free accepted manuscript: https://doi.org/10.35542/osf.io/p2dvf_v1

Data: IPEDS 150%-time graduation rates, 2018 to 2022, via the Urban Institute Education Data Portal; cohorts of at least 30 in each group. The article’s data availability statement points to the full replication materials.

Related, on how the choice of gap measure changes what you see: Effect sizes for education: what WWC actually wants. I build baselinr and am developing a cohort course on credible evaluation and measurement in education. Subscribe via RSS to follow along.

A trial is only randomized on day one

Random assignment buys you two groups that are alike, once, at the start. Every departure after that is chosen, by a family that moved, a student who stopped coming, a school that quit the study, and chosen departures re-sort the groups. If the program’s strugglers give up on the study while the control group’s strugglers have nothing to give up on, the surviving program group is now selected for persistence in a way its comparison is not. This is the enrollment problem sneaking back in through the exit door, after randomization was supposed to have solved it.

That is why attrition is not a footnote about sample size. The evidence standards used to review education RCTs treat it as the first gate: the What Works Clearinghouse rates a randomized trial down if attrition is high, before anything else about the analysis is considered.

Two numbers, and the quieter one matters too

Attrition comes in two flavors, and reports love to disclose the first while staying vague on the second. Overall attrition is how many outcome measurements went missing in total. Differential attrition is the gap between the groups, and it is the louder alarm, because it means whatever drove students out operated differently on the two sides.

But equal rates are not automatic safety. Ten percent can leave each group for different reasons, the discouraged leaving one arm and the upwardly mobile leaving the other, and the comparison bends with no differential attrition at all. The rate is observable; the reasons mostly aren’t. Which is why the honest report does more than disclose percentages.

What the careful ones show

Three things, none of them exotic. A flow diagram: how many were assigned, served, measured, in each group, so the reader isn’t reconstructing it from footnotes. A leavers-versus-stayers comparison: the same baseline table the study already built, rerun on who dropped out, so you can see whether the missing tenth was the struggling tenth. And an analysis that keeps everyone assigned in the group they were assigned to, with a sensitivity check that asks how bad the missing outcomes would have to be to change the conclusion.

When a report instead analyzes “completers,” praises their gains, and moves on, question three of my reading checklist has been answered, just not in words. I say this with sympathy; a completers analysis is usually the easiest query to write, and it is easy to forget that it answers a different question.

The thirty-second version

Find the number randomized and the number analyzed, per group; the distance between them is where the story lives. Ask whether the leavers looked like the stayers at baseline. And check whether the headline number is about everyone who started or only those who finished, because “the program worked for the students who stayed” is sometimes the program’s success and sometimes its selection mechanism, and the report’s job is to help you tell which.

I build baselinr, a small R package for baseline-equivalence tables, including the leavers-versus-stayers kind, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

Why such a small thing does so much

Almost none of what pre-registration prevents is fraud. It prevents something more human: hopeful flexibility. Every evaluation contains dozens of defensible forks (which outcome, which subgroup, which model, which sample cutoff) and a researcher who wants the program to work can wander those forks in good faith until something significant appears. I wrote about the reader’s side of this, the subgroup safari and the migrating outcome, and about the two questions that expose it. Pre-registration is the author’s side: you close the forks yourself, in advance, while you still don’t know which answer flatters you.

The first time I tried to write a registration for a small course study, the embarrassing part was discovering how many decisions I had been planning to make “later,” which is to say, after peeking. Closing them in advance took an afternoon and felt like flossing: unglamorous, obviously correct.

It also, quietly, upgrades every null you’ll ever report. A null against a pre-named target is a finding. The same null without the registration is an anecdote that invites the question “but did you check…?”

What it doesn’t do

A registration is a plan, not a cage, and it is worth saying what it can’t carry. It doesn’t make a weak design strong; a pre-registered comparison with no comparison group is still a description. Deviating from the plan is fine, science requires it, so long as the deviation is disclosed as one. And exploratory analysis remains welcome, labeled as exploration; the registry is what makes the label meaningful.

Education has the infrastructure now

Medicine had a twenty-year head start, but education is no longer short of places to do this. The Registry of Efficacy and Effectiveness Studies, built by SREE and ICPSR specifically for education research, takes randomized trials, discontinuity designs, and quasi-experiments alike; OSF Registrations covers everything else. Funders and evidence reviews increasingly ask for it. The cost is real but small: for a typical evaluation, the registration is an afternoon.

How I use one when reading

Three steps, thirty seconds each. Find the registration link (its absence is itself information). Check the date against when outcome data could first have been seen. Compare the registered primary outcome to the one in the abstract. When all three line up, most of the darker explanations for a striking result are simply off the table, and I can spend my skepticism on the design instead.

For anyone running an evaluation this year, from a grad student with her first studies still ahead of her: the two hours it takes are the cheapest credibility you will ever buy. You are not tying your hands. You are making it possible, later, for people like me to believe you.

I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

First: is the null informative, or just imprecise?

“No detectable effect” covers two different findings. A tight confidence interval around zero says something real: effects big enough to care about probably aren’t there. A wide interval mostly says the study couldn’t answer the question, which is what happened in many of those 141 trials. The way to tell them apart is to translate the interval into units a reader can feel: “we can rule out gains larger than a month of learning” and “we can’t rule out anything between harm and half a year of gains” deserve opposite reactions.

Second: could the study have hidden a real effect?

Before concluding the program failed, I check whether the design could have produced a null even if the program worked.

• The outcome couldn’t see the change. An unreliable or off-target measure shrinks every effect toward zero, which is why I check measurement before models, and why an outcome that was itself machine-classified with error attenuates further.
• The score doesn’t mean the same thing in both groups. A null gap on a scale that functions differently across groups is not evidence of equality, in either direction.
• The groups weren’t comparable to start. If baseline equivalence failed, a real gain in one group can be canceled by a head start in the other. The same worry applies to matched comparisons, where the invisible reasons people enrolled can mask or mimic effects.
• The outcome was aggregated away. Composites hide movement; what a summary score actually captures and how the weights were set both decide whether a real change surfaces.

Third: is the null the honest number, at last?

The mirror image matters just as much. Analyses that treat clustered students as independent produce stars out of noise, and piling on control variables can manufacture effects as easily as destroy them. Sometimes a well-analyzed null is not the study failing; it is the first analysis in the room that stopped flattering the program.

Then write it like a result, because it is one

Lead with the finding, in the first paragraph, in plain words. State what the interval rules out, not just the p-value. Point to what success was defined as in advance, if anything was, since a null against a pre-named target is a result while a null against nothing is an anecdote. Keep subgroup patterns, labeled as exploratory, for the next study to test. And separate the two sentences a decision-maker actually needs: “it probably doesn’t work” and “we couldn’t tell,” one of which should appear, alone, in your summary. If you want to see these habits from the reader’s side, they are questions seven and ten on my checklist.

I’m still in grad school; nobody has yet paid me to deliver bad news to a funder, and I don’t imagine the room feels like a statistics seminar. But a clean null saves someone real money, usually the district about to scale the program. And it compounds: people who watch you report a null clearly will believe your positive finding next year at face value. In a field where the average large trial finds 0.06 standard deviations, the evaluators who can say “nothing happened” gracefully are not the failures. They are the ones telling the truth at the base rate.

I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

What matching actually does

Matching does one thing, and does it honestly: it makes the comparison group resemble the participants on the variables you matched. Same prior grades, same attendance band, same school poverty level. That is real progress; an unmatched comparison would be worse. After matching, the observable excuses are gone.

What it cannot do

The trouble is that the reason a student enrolls is usually not a column in the file. A tutoring program’s sign-ups differ from non-sign-ups in motivation, in whether a parent pushed, in whether a counselor happened to mention it. Matching on grades cannot touch any of that, because none of it is recorded. The technical phrase is “selection on unobservables”; the plain version is that matching removes the differences you can see, and the decision to participate is driven partly by differences you can’t.

There is a subtler version of the trap that took me longer to see. Match a signed-up student with low prior grades to a non-participant with the same low grades, and you may have paired the most motivated student in one group (low grades, still sought help) with an ordinary student in the other. Matching on the observable can quietly sharpen the unobservable difference, not shrink it.

The four things I look for in a matched study

A balance table, before and after. Matching is a claim about achieved similarity, and the claim is checkable: the same baseline-equivalence table any comparison-group study owes, shown for the matched sample. No table, no claim.

The enrollment story. I try to write one honest sentence about how a student ended up in the program. If the sentence contains a word like “chose,” “volunteered,” or “was referred,” I ask whether two students with identical files could plausibly differ on exactly that word. Usually they could.

Who got dropped. Matching discards participants with no good counterpart, which changes the question from “did the program work?” to “did it work for the kind of participant who has a look-alike?” A report should say how many were dropped and who they were.

A sensitivity analysis. The best matched studies ask: how strong would a hidden difference have to be to erase this effect? Reporting that number doesn’t remove the assumption; it prices it. When it’s missing, I price it myself, pessimistically.

The right amount of trust

None of this makes matching a trick. It is an honest tool for a narrower claim than the one usually attached to it, something like: among students who look alike on what we measured, participants did better. Studies that state the claim at that size, show the balance, and price the assumption are the ones I trust most. The reassuring sentence at the top was never the evidence. The table under it is.

I build baselinr, a small R package for exactly that balance table (WWC-aligned, before and after matching), and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

What a checklist can and can’t do

Ten questions can’t convict a report, and passing them doesn’t make a finding true. Most of them cover mistakes that are easy to make and easy to miss when reading. This is triage: it tells me where to spend skepticism, which is always in shorter supply than I’d like, and it protects me a little from my own preferences. The reports I end up trusting most are rarely the ones with the biggest effects. They are the ones that make questions 1 through 10 easy to answer, because the authors asked themselves first.

I build baselinr, a small R package for the baseline-equivalence table in question 2, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

What surprised me

I expected some reshuffling in the middle. I did not expect the list to nearly turn over. UW Seattle is first when the weight is on graduation (84% of its students finish, the highest here) and last when the weight is on Pell share (17% of its undergraduates receive Pell, the lowest here). Evergreen State is the exact mirror: last on one end, first on the other. These aren’t two rankings of quality and its absence. They are two different questions, and each school answers one of them well.

The reason the reversal is so clean is worth stating: in this group of eight, the two measures are strongly negatively related. The campuses that graduate the largest share of students enroll the smallest share of Pell students. I don’t get to conclude much from eight schools about why that is, but it has an uncomfortable implication for anyone building a rating: there is no weight between these two numbers that is neutral. Every blend takes a side, quietly.

What this little exercise can’t say

A lot, honestly. Two indicators is a cartoon; real rating systems use many, and better ones. A graduation rate reflects who enrolls at least as much as what a campus does for them. Pell share is a coarse stand-in for access. And my choices leaked in everywhere: I checked the endpoints of the figure under two other reasonable ways of scaling the numbers and they came out the same, but the 50/50 column did not. Even the scaling is a decision.

Above all, this is not a ranking of Washington’s universities, and none of these schools is “worst.” That is closer to being the point: any single ordered list of them would be hiding a judgment call in the weights.

What I do differently now

When I see a composite rating, I try to find two things before reacting to any school’s position: what is weighted how much, and whether the publisher shows how the order moves under other defensible weightings. When that sensitivity is reported, I trust the list more, not less.

I’m working on audits of real rating and flagging systems with public data, and I’ll write them up here as they become shareable.

Update, October 2026: the first of these is out. If colleges were employers, two-thirds of those we can audit would fail the government’s own fairness test.

The data (a small CSV pulled from the Urban Institute’s public IPEDS API) and the figure script (make_figure.py) are in this post’s folder; deleting the CSV and rerunning the script re-fetches the numbers.

Related: What a “teacher effectiveness” score actually measures (and what it doesn’t). I build baselinr and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

Three reasons a variable is in your model

One: it’s a confounder. Something that causes both who got treated and the outcome. Pretest scores, if the higher scoring kids were also more likely to get the program. You want these in the model. Leaving them out is the thing everyone’s (rightly) scared of.

Two: it’s a mediator. Something the treatment causes, which then moves the outcome. It’s part of how the program works. Control for it and you’re literally subtracting off a piece of the effect you came to measure.

Three: it’s a collider. The weird one. It’s a shared consequence of two things, and it sits there harmlessly until you condition on it, at which point it invents a correlation that was never real.

“Control for everything you can” treats all three the same. Two of them make your answer worse.

Just draw the arrows

This is less philosophical than it sounds. Sketch what causes what (a DAG, if we’re being formal). Then the rule is basically mechanical: block the paths that sneak in through the treatment, and leave alone anything the treatment causes or anything two variables jointly cause.

A reviewer saying “control for X” is making a guess about where X lives on that picture. It’s not a command. Go look at where it actually sits.

The one that got me

Say you’re evaluating a tutoring program and your outcome is test scores. Being responsible, you decide to control for student motivation. Reasonable, except you measured motivation after the program ran. Which means the program probably moved it. So now motivation is downstream of the treatment, and controlling for it shaves off exactly the part of the effect that went “kids got more into it, then did better.” You conclude tutoring barely did anything.

It did something. You just deleted the evidence. And it felt like diligence the whole time, because, again, the coefficient moved.

Watch it happen

I simulated this so it’s not just me asserting things. True total effect of the program: 8.5. There’s a trait C that drives both treatment and the outcome (a real confounder), and M is motivation measured after the program (a mediator).

# Estimand: total effect of `treat` on `Y` (simulated; true value = 8.5)
coef(lm(Y ~ treat))["treat"]          # naive
coef(lm(Y ~ treat + C))["treat"]      # adjust for the confounder
coef(lm(Y ~ treat + C + M))["treat"]  # also adjust for the post treatment mediator

The naive number is too big because treated kids started ahead. Adjust for the confounder and it snaps to 8.61, basically the truth. Then you add the after the fact mediator and it falls to 3.99, because you just controlled away the thing the program works through. The model with the most controls is the most wrong. That’s the whole point.

What I actually do now

• Sketch the DAG before I touch the model. A bad drawing on a receipt counts.
• Only control for things measured before treatment.
• If a variable was measured after the treatment, I treat it as off limits until the picture tells me otherwise.
• For every control, I make myself say which of the three it is. If I can’t, it doesn’t go in.

I’m not against controls. I’m anti “more controls = more rigor,” because that one’s just false, and it’s the kind of false that looks like care.

I’m building baselinr, a small R package for WWC-aligned baseline equivalence, and a cohort course on running education evaluations you can actually trust. subscribe via RSS if that’s your thing.

What a noisy outcome quietly does

Random measurement error in your outcome doesn’t just scatter the points around a bit. It inflates the variance of the outcome, which widens your standard errors, which costs you power. And because the standard deviation in your effect size denominator just got bigger, the standardized effect you can report shrinks. You can run a flawless design on a bad measure and find nothing, not because the program failed, but because your ruler was made of rubber.

So before the model, check the ruler.

Reliability: what one number hides

Reliability usually shows up as a single coefficient, Cronbach’s alpha most of the time, and gets a nod on the way to the regression. But that one number is hiding things. Alpha climbs with the number of items, not only their quality, so a long mediocre scale can post a comfortable one. It leans on the items being roughly interchangeable, and when they aren’t it can mislead in either direction (which is why a lot of measurement people reach for McDonald’s omega instead). And a high alpha is not proof your items measure one thing.

A “respectable” reliability can also just be too low for your claim. Detecting a small effect on a moderately unreliable outcome can demand far more sample than you have. The reliability you need is a function of the effect you’re chasing, not a fixed bar you clear once and forget.

Validity, briefly

Reliability is consistency. Validity is aboutness. A bathroom scale that reads three pounds heavy every single time is perfectly reliable and perfectly wrong. For an outcome, the evidence worth showing is whether the items cover the construct, whether the data back the dimensions you claim, and whether the measure relates to things it should and not to things it shouldn’t. You don’t need all of it for every study. You do need to stop claiming more than your evidence supports.

A check you can actually run

Before you trust an outcome, look at it. On your own data, not invented numbers, look at the item behavior and an internal consistency estimate for each key construct:

# On YOUR data. Do not report invented coefficients.
# items <- your_data[, c("q1", "q2", "q3", "q4")]
# psych::alpha(items)       # internal consistency + item level diagnostics
# psych::omega(items)       # model based reliability
# psych::fa.parallel(items) # how many dimensions are actually in there?

Read the item level output, not just the headline. An item that barely correlates with the rest, or a “scale” that turns out to be two factors in a trench coat, will tell you more than alpha ever does alone.

Why this goes before the impact model

Because everything downstream inherits it. Here’s the same point, simulated. The program has a real effect, true standardized 0.5. The only thing I change is how reliably the outcome is measured:

Same program. Same design. Same sample. Only the ruler gets noisier, and the effect you’d report slides from 0.50 to 0.31 while your power to see anything falls from 70% to 34%. Half your ability to detect a real effect, gone, because the outcome measure was weak.

Measurement isn’t the boring prelude to the real analysis. On a bad measure, it is the result. Check the ruler before you trust the measurement.

I build baselinr and am developing a cohort course on credible evaluation and measurement in education, where this is basically a whole week. subscribe via RSS to follow along.

What it’s trying to do

A value added model starts from a fair idea. Predict each kid’s score from where they started and a few covariates, and credit the teacher with the part that beat the prediction. Net out the head start, reward the gain. I don’t think the goal is wrong.

The problem is that the teacher’s real contribution is small next to everything else shoving those scores around, and the one thing the model can’t fully scrub out is the biggest one: kids are not handed to teachers at random.

What’s actually inside the score

The number says “effectiveness,” but it’s carrying a lot of passengers. Some of it is real teaching. Some of it is sorting, because if the stronger or better supported kids keep landing with certain teachers, adjusting for prior scores shrinks that bias but never kills it, and whatever you didn’t measure rides straight through. Some of it is test measurement error, which goes in one end and comes out the other as “effectiveness.” Some of it is just small samples, a teacher judged on one or two classrooms, where a couple of unusual kids swing the whole thing. And some of it is the test’s scale.

Stack all of that up and the year to year stability of these scores tends to be modest, well below what a high stakes ranking quietly assumes.

Watch it happen

I gave 300 simulated teachers a fixed, unchanging true effectiveness. It does not move. The only thing that changes between year one and year two is noise: small classes, test error, the luck of the draw.

vam1 <- mu + noise   # year 1 estimate
vam2 <- mu + noise   # year 2 estimate, same true mu, fresh noise
cor(vam1, vam2)      # year to year reliability

The year to year reliability comes out at 0.47. Of the teachers in the bottom quartile one year, 23% land above the median the next, same people, same skill. And a representative teacher slides from the 42nd to the 69th percentile between years without changing a thing.

That jump from the low 40s to the high 60s, the one a district would frame as a turnaround? Pure noise. The scatter says it flatly: where a teacher ranks one year barely tells you where they’ll rank the next.

And then it gets worse

Once a number carries stakes, Campbell’s Law shows up: the moment a metric becomes a target, people optimize the metric instead of the thing it was supposed to stand for. Teaching to the test. Quietly steering which kids end up in your room. Narrowing what gets taught. The score stops being even the noisy proxy it started as.

What to do

If you’re going to use these at all: report the uncertainty, and actually look at how wide it is. Shrink the estimates (empirical Bayes pulls noisy small sample numbers toward the average, which is exactly right when most of the spread is noise). Never hang a high stakes decision on a single year. And say out loud what you’re assuming about how kids got sorted to teachers, because the entire causal story rests on it.

These scores can inform. Used as a one number verdict on someone’s career, they mostly measure which children walked through the door.

This is where methods and policy collide, which is the part of this work I care about most. I build baselinr and a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

The thing itself

Strip it down: take the gap between the two group means, divide by a standard deviation. Now the effect is in standard deviation units, so a reading program and a math program can sit on the same ruler. Two studies with the same raw point gain can land at very different standardized effects, because the spread of scores differs. Which is the point. It’s also where the choices start to matter.

Cohen’s d vs. Hedges’ g

Cohen’s d puts the pooled within group SD in that denominator. Fine, except it has a known flaw: in small samples it’s biased upward. It overstates. And education studies are small constantly, a few classrooms, a handful of schools, so this is not a hypothetical.

Hedges’ g is just Cohen’s d times a small sample correction. When the sample is big, the correction is basically 1 and the two agree. When it’s small, the correction pulls the estimate back toward the truth. WWC reports g for exactly this reason, and once you see the size of the bias you stop arguing about it.

Here’s the bias, simulated. True standardized effect is 0.5. I draw a pile of small studies and average each estimator:

Three per group and uncorrected d averages 0.618, a 24% overstatement of an effect that is really 0.5. Hedges’ g sits at 0.495. By a hundred per group they shake hands. Small education studies live on the left edge of that plot, which is precisely where skipping the correction costs you.

The little decisions people get wrong

Which SD goes in the denominator. Pooled (both groups) is the usual choice and what the WWC convention assumes. control group only is sometimes defensible, but if you pick it without saying so, your effect size isn’t comparable to anyone else’s. State your denominator.

Treating a small sample d as if it were unbiased. If n is small and you skipped the correction, your number is inflated. Use g.

And comparing effect sizes across studies that standardized differently. A 0.30 on a narrow, homogeneous sample is not the same animal as a 0.30 on a broad one. Effect sizes are only comparable when the rulers are.

Reporting it like you mean it

Give the effect size with an interval, not a lonely point estimate; “g = 0.8” with no uncertainty begs to be over trusted. Tie the number to a decision, because “significant” and “meaningful” are different words. And say your choices out loud so the number can be checked.

A standardized mean difference is a tiny calculation carrying a big claim. Get the denominator and the correction right, attach honest uncertainty, translate it into something a human can act on. Skip those and “significant” becomes a word that hides more than it tells.

I build baselinr, a small R package for WWC-aligned baseline equivalence and effect sizes, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

The question it’s actually answering

Plain version: before the program started, were these groups similar enough that I can pin later differences on the program instead of on who was in each group?

Notice what that is not. It is not “is the baseline difference statistically significant?” That’s the reflex, run a t test on the pretest, see p > .05, declare the groups fine, move on. But significance is the wrong tool here, for a reason that has nothing to do with the program. With a big sample, a trivial difference turns “significant.” With a small one, a real gap sails through as “not significant.” A p value is mostly telling you about your sample size. Equivalence is a question about size of the gap. Different question.

What the WWC asks for instead

For each covariate you report a standardized mean difference (Hedges’ g, which is Cohen’s d with a small sample correction), and you read it against two numbers:

The whole framework is those two numbers, 0.05 and 0.25, applied honestly. The arithmetic isn’t the hard part. Remembering what each category obligates you to do later is.

Doing it without a spreadsheet

Here it is on a simulated tutoring evaluation, 400 students, data I bundled into baselinr just for this. One call, whole table:

library(baselinr)
data(tutoring)

baseline_equivalence(
  tutoring, treatment = "treat",
  covariates = c("pretest", "attendance", "age", "female", "frpl", "ell")
)

Continuous covariates get Hedges’ g; the binary ones get the Cox index, handled for you. And honestly the plot is where it clicks:

love_plot(baseline_equivalence(tutoring, "treat",
          covariates = c("pretest", "attendance", "age", "female", "frpl", "ell")))

Age and sex are equivalent, nothing to do. Pretest and attendance sit in the middle band, so you can proceed but you owe the model an adjustment. And then poverty status and language status blow past 0.25, which means the groups differ on two things adjustment can’t fix. That last pair is something you’d want to know before you ever report an effect, not after a reviewer finds it.

The part that quietly leaks credibility

“Satisfied with adjustment” is not a passing grade. It’s a promise. It says: these groups are close enough, as long as I actually control for this covariate in the impact model. And it is genuinely common to see a baseline table that files a covariate under “satisfied with adjustment” and then an impact model that never adjusts for it. The table says one thing, the model does another, and a careful reader catches it instantly. If a covariate lands in the middle band, it has to show up in the model. Every time.

Treat the baseline table like the load bearing wall it is, not the trim you tack on at the end. Get it right and everything downstream has something solid to stand on. Get it wrong and your fanciest model is just measuring the head start.

I build baselinr, a small R package that produces these tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.

The lens thing

I think of a scale as a lens you look at a construct through. Comparing two groups’ scores assumes they’re looking through the same lens. If the lenses are ground differently, then comparing what each group “sees” tells you about the lenses, not the thing you were trying to look at. The technical name for “same lens” is measurement invariance, and it is not a given. It’s a thing you check.

How much you’re allowed to say

Invariance comes in levels, and each one buys you a different comparison. Roughly: if the same items load on the same factor in both groups (configural), you’ve got the basic shape. If the loadings are equal too (metric), you can start comparing relationships across groups. And only if the intercepts are also equal (scalar) are you actually allowed to compare means.

That last one is the level your group comparison quietly assumed it had. You test for it with a multigroup CFA. And when a single item behaves differently across groups, that’s DIF, differential item functioning, which is a fancy way of saying the lens has a chip in it right there.

Watch it happen

Two groups, a 5-item engagement scale, and here’s the thing: I built them to have identical true engagement. Same distribution, no real difference. Then I made one item (item 5) read systematically lower for group B at the same level of engagement. Textbook DIF. Then I just… compared the groups, the way anyone would, with a standardized mean difference:

library(baselinr)
hedges_g(theta, group)        # the true latent difference
hedges_g(scale_full, group)   # observed, all 5 items
hedges_g(scale_inv, group)    # observed, dropping the one DIF item

The groups are equal. I promise, I made them that way. And the full scale still reports g = −0.25, which most people would write up as “Group B is less engaged.” The figure shows the crime scene: items 1 through 4 are flat, and item 5 is doing all of the work, off by −1.22 on its own. Pull that one item and the difference evaporates to −0.02.

Nothing about engagement happened here. The ruler happened.

What to do

Test invariance before you compare means, every time you’re comparing a latent thing across groups. If it fails, don’t throw the whole comparison out, go find the items that broke (the DIF ones). And then report partial invariance honestly, anchored on the items that held. It’s less clean than a single triumphant p value. It’s also the difference between a finding about students and a finding about your questionnaire.

Measurement is the thing I get genuinely nerdy about. I build baselinr and I’m building a cohort course on credible evaluation and measurement in education. subscribe via RSS if you want in.

Why “n = 3,000” is fiction

Kids in the same school resemble each other. Same teachers, same building, same neighborhood, same everything. So each extra student from a school you’ve already sampled tells you less than a brand new, unrelated student would. Regression assumes your rows are independent. When they’re clustered and you shrug and run it anyway, your standard errors come out too small, your test statistics too big, and out the other end pours a parade of “significant” effects that are mostly just the model fooling itself.

How much smaller depends on the intraclass correlation, the ICC, which is the share of the variation that lives between schools rather than within them. A little clustering barely dents you. A lot of it quietly guts your real sample size down toward the number of schools rather than the number of students.

The machinery, quickly

ICC measures how much students within a cluster move together. The design effect turns that into how badly your naive standard errors lie. Cluster robust standard errors and multilevel models are the two standard fixes (robust SEs patch the uncertainty; multilevel models patch it and let you study how schools differ). And if you’ve only got a handful of clusters, even robust SEs get shaky, and you reach for a wild cluster bootstrap.

Watch it happen

This is my favorite part, because it’s so stark. Sixty schools, fifty kids each, a school level treatment, and a true effect of exactly zero. The within school correlation gives an ICC around 0.26, a design effect of roughly 14, so those 3,000 students carry about as much information as 210.

One representative null study:

m <- lm(y ~ treat, data = study)        # student level, ignores clustering
sqrt(diag(vcov(m)))["treat"]                          # naive SE
sqrt(diag(sandwich::vcovCL(m, cluster = ~school)))["treat"]  # cluster robust SE

Same estimate, same data. The naive version is screaming significance. The honest version shrugs, correctly, because there is nothing there.

And it’s not one unlucky draw. Across 600 simulated null studies:

The naive analysis declares a significant effect in 57% of studies where the truth is nothing. 57 percent. The cluster robust version sits at 6%, right about the 5% you signed up for. If you ignore clustering, “p < .05” doesn’t mean what you think it means. It barely means anything.

What to do

Cluster at the level treatment was actually assigned (rolled out by school? then school, not student). Report the ICC so your reader knows how much this even matters here. Watch out when you’ve only got a few clusters. And if you genuinely care about how schools differ, not just correcting for it, go multilevel.

Before you celebrate a tiny p value, just ask where the rows came from. In education they’re almost always nested, which means “n” is not the number of students, and significance computed as if it were is significance you didn’t earn.

I build baselinr and I’m building a cohort course on running education evaluations you can actually trust. subscribe via RSS to follow along.

A predicted label is a guess you’re treating as a fact

That’s the whole problem in one sentence. You didn’t observe “this feedback is specific.” You estimated it, with a model that’s right most of the time and wrong some of the time. The minute you drop those predictions into a regression as if they were the real thing, you’ve imported the model’s mistakes without telling the regression they’re there.

And the phrase “ground truth” makes it so much worse, because it’s not ground truth. It’s the classifier’s training target. In your actual analysis those labels are a noisy stand in, and calling them truth is how the error sneaks past you.

What the error does depends on what kind it is

If the mistakes are just random (the classifier is equally clumsy everywhere), and the noisy variable is a predictor, the effect gets attenuated. Pulled toward zero. This is the part people get wrong in conversation, by the way. “It’s just noise, it averages out.” A regression coefficient on a mismeasured predictor does not average out. It shrinks.

Differential error is the one that actually scares me. If the classifier is worse for some cases than others, in a way that’s tied to your outcome or your groups, the bias picks up a direction. You’re not just underestimating anymore. You can end up pointing the wrong way.

Your setup is the textbook case: you predict “feedback specificity” from teacher comments, then relate it to student progress. Now suppose the model reads feedback from teachers of multilingual learners a little worse. That error is now braided into exactly the comparison you care about, and it doesn’t wash out. It bends the conclusion.

Watch it happen

Six thousand simulated comments. The true effect of the text feature on the outcome is 1.0. I feed two classifiers’ predicted labels into the same regression and treat them as truth:

# True effect of Z on Y is 1.0 (simulated).
coef(lm(Y ~ Z))["Z"]         # if we had the true labels
coef(lm(Y ~ Zhat_nd))        # 85% accurate, error unrelated to anything
coef(lm(Y ~ Zhat_d))         # 85% accurate overall, but worse for one group

Sit with two things here. First: clean, random, 85%-accurate error drags a true effect of 1.0 down to 0.68. A third of your signal, gone, purely from treating guesses as data. Second, and this is the one that gets me: both classifiers report the same 85%. But the “differential” one is 93% accurate for one group and 78% for the other. The headline number cannot tell you which world you’re in. Only breaking accuracy out by subgroup can.

What to actually do

Keep a human labeled gold set, and not just for tuning. You need it to measure the error so you can correct for it later. Report classifier performance broken out by the groups you’ll compare, because “85% overall” is precisely where differential error hides. And then either propagate that uncertainty into the analysis or use an estimator built for predicted labels (there’s a small but real literature on inference with machine predicted variables now). Whatever you do, stop quietly swapping predictions in for observations and calling it done.

Measure your measurement. Report it by group. Let the error into the model the same way it’s already, invisibly, in your data.

This exact corner, where text models meet measurement error, is what I work on. I build baselinr and I’m putting together a cohort course on credible evaluation and measurement in education. subscribe via RSS if you want to follow along.