Standardised test scores have long been treated as the end-all-be-all of education. Researchers and the public at large generally use math and reading test scores to gauge education quality across countries, schools, and interventions. But a growing body of empirical evidence suggests that we have probably gotten it all wrong. Here’s why. 

A study recently released by the American Enterprise Institute found that standardised test scores are weak predictors of long-term success. Specifically, the authors collected 34 studies that evaluated the effects of school choice programs on both test scores and high school graduation. The study found that 61 percent of the effects on math test scores — and 50 percent of the effects on reading test scores — did not successfully predict effects on high school graduation. Similarly large divergences were found between choice programs’ effects on student test scores and their effects on college enrollment.

But that’s not all. Standardised test scores do not appear to be strong predictors of other long-term outcomes either. I have compiled more evidence of these divergences that exist in the most-rigorous private school choice literature. My search revealed 11 disconnects between private schools’ effects on test scores and their effects on other arguably more important educational outcomes.

For example, an experimental evaluation of a private school voucher program in Washington, D.C. found that winning the lottery to attend a private school had mixed effects on test scores, but over a 50 percent increase on students’ tolerance of others. Another rigorous evaluation of a voucher program in Milwaukee found no effects on reading test scores after four years. On the other side of the equation, my coauthor and I followed the same sample of students until they were around 22 to 25 years old and found that the Milwaukee program reduced the likelihood that they committed crimes as adults by over 50 percent. Other studies found significant divergences between test scores and outcomes such as charitable giving, political participation, effort, and happiness in school.

And divergences exist outside of the school choice literature as well. At least five studies that I know of using rigorous value-added methodology find disconnects between teachers’ effects on student test scores and their effects on student character skills such as behavior and effort. For example, Northwestern University professor Kirabo Jackson finds that teachers’ effects on student behavior are much stronger predictors of high school graduation than their effects on student test scores. Indeed, Jackson finds that teachers’ effects on student behavior are over 8 times as influential for high school graduation as effects on student test scores.

Of course, we cannot ignore the fact that some studies do find a link between test scores and long-term outcomes. For instance, prominent education scholars – Chetty, Friedman, and Rockoff — found that teachers that improve student test scores also tend to have positive effects on earnings later on in life. But that does not at all mean that their effects on test scores caused the effects on earnings. It is more likely that — in their sample — teachers that were good at shaping test scores were also good at shaping the non-cognitive skills necessary for success in the long-run, on average.

In other words, teachers that are good at improving standardised test scores can also be good at motivating students to work hard and to treat others with respect. Hard work and respect may be the skills that influence long-run outcomes such as earnings. But it is nearly impossible to accurately measure soft skills, which could be why we have found so many divergences in the literature.

Either way, what appears clear is that focusing too much on test scores could compromise the character development necessary for true lifelong success.