The two hours that make an evaluation believable
The most convincing before-and-after in the history of research credibility took two decades to show up in the data. Large heart-disease trials funded by one US agency were, until 2000, positive more often than not: 17 of 30 published trials, 57%, reported significant benefits. Then a rule changed. Trials had to register their primary outcome, in public, before the data came in. Of the 25 trials published after the rule, 2 reported significant benefits. Eight percent.
The drugs didn’t get worse in 2000. What changed is that researchers could no longer decide, after seeing the data, which outcome the study had “really” been about. That is the entire idea of pre-registration: a dated, public note, written before the outcome data exist, saying what will be measured, how it will be analyzed, and what would count as success.
Why such a small thing does so much
Almost none of what pre-registration prevents is fraud. It prevents something more human: hopeful flexibility. Every evaluation contains dozens of defensible forks (which outcome, which subgroup, which model, which sample cutoff) and a researcher who wants the program to work can wander those forks in good faith until something significant appears. I wrote about the reader’s side of this, the subgroup safari and the migrating outcome, and about the two questions that expose it. Pre-registration is the author’s side: you close the forks yourself, in advance, while you still don’t know which answer flatters you.
The first time I tried to write a registration for a small course study, the embarrassing part was discovering how many decisions I had been planning to make “later,” which is to say, after peeking. Closing them in advance took an afternoon and felt like flossing: unglamorous, obviously correct.
It also, quietly, upgrades every null you’ll ever report. A null against a pre-named target is a finding. The same null without the registration is an anecdote that invites the question “but did you check…?”
What it doesn’t do
A registration is a plan, not a cage, and it is worth saying what it can’t carry. It doesn’t make a weak design strong; a pre-registered comparison with no comparison group is still a description. Deviating from the plan is fine, science requires it, so long as the deviation is disclosed as one. And exploratory analysis remains welcome, labeled as exploration; the registry is what makes the label meaningful.
Education has the infrastructure now
Medicine had a twenty-year head start, but education is no longer short of places to do this. The Registry of Efficacy and Effectiveness Studies, built by SREE and ICPSR specifically for education research, takes randomized trials, discontinuity designs, and quasi-experiments alike; OSF Registrations covers everything else. Funders and evidence reviews increasingly ask for it. The cost is real but small: for a typical evaluation, the registration is an afternoon.
How I use one when reading
Three steps, thirty seconds each. Find the registration link (its absence is itself information). Check the date against when outcome data could first have been seen. Compare the registered primary outcome to the one in the abstract. When all three line up, most of the darker explanations for a striking result are simply off the table, and I can spend my skepticism on the design instead.
For anyone running an evaluation this year, from a grad student with her first studies still ahead of her: the two hours it takes are the cheapest credibility you will ever buy. You are not tying your hands. You are making it possible, later, for people like me to believe you.
I build baselinr, a small R package for baseline-equivalence tables, and am developing a cohort course on credible evaluation and measurement in education. subscribe via RSS to follow along.