One might expect that a report with a title referring to “more findings about school vouchers and test scores” would include the latest studies. However, Mark Dynarski, a Senior Fellow at the Brookings Institution, and Austin Nichols, a Principal Associate of Social and Economic Policy at Abt Associates, recently argued that the new voucher evidence is negative, even though the two most recent studies in Louisiana and Indiana show null to positive effects on student achievement.

How did these researchers come to their counterintuitive conclusion?

It appears that the researchers did not include the two most recent evaluations of voucher programs in Figure 1 below, which is taken from their report. That would not be much of a problem if the title and body of the report did not indicate otherwise.

Figure 1: Findings from Four Current Studies of Vouchers and Eight Previous Studies

If the researchers simply used previous results, the article (and its title) should not indicate that the latest estimates from Indiana and Louisiana are used. Further, the forest plots should not indicate that multiple years of results are used.

So what does the experimental voucher research actually say?

The Actual Test Score Results – and Their Implications

Even the latest experimental results, which show that voucher students in Louisiana and Indiana caught up with or did better than their public school peers on test scores, are less optimistic than prior voucher studies. However, there is not a clear theory for why voucher programs ought to be less-effective now than they used to be, all else equal.

I suspect that the regulatory environment may have something to do with the recent lackluster experimental results. For example, private schools participating in the Louisiana Scholarship Program (LSP) must administer the state standardized test, prohibit parental copay for families using vouchers, report finances to the government, and surrender their admissions process over to the state. As the recent study by me and my colleagues at the University of Arkansas finds, only a third of the private schools chose to participate in the LSP, and those schools were less likely to be the higher quality institutions.

As shown in Figure 2 below, the recent meta-analysis of 19 experimental voucher studies, by researchers at the University of Arkansas, finds that student test score impacts improve over time (also shown by the most recent results in Louisiana and Indiana). Indeed, students using a voucher to attend a private school have around half a standard deviation higher academic achievement in reading than their traditional public school peers by the fourth year. That is quite significant; a study by Stanford economist Eric Hanushek indicates that the effect is equivalent to around a 6.5 percent increase in lifetime earnings—more than $75,000.

Figure 2: Experimental Impacts of Vouchers on Student Test Scores by Year

While the positive test score trend is encouraging, I am skeptical about the implications. The trend could indicate that children are settling into their new schools and that private schools are improving in academic quality over time. However, the trend could also simply be a reflection of the new incentive structure for private schools in voucher programs. Since test scores are the state’s preferred accountability measure, private schools in voucher programs must shift their focus towards standardized testing.

The latter explanation could be problematic. As Jay Greene from the University of Arkansas has pointed out, there are nine school choice studies showing a disconnect between test scores and long term outcomes that we actually care about—such as graduation rates, criminal activity, and income. The shift towards focusing more on standardized testing could very well harm the ability of private schools to shape the character skills necessary for success.

If we really want to help all children achieve the best education possible, we must be humble about our ability to measure progress and determine what is best for people who have unique needs. Rather than using a single crude measure to determine who ought to have access to more educational options and a better life, we should have faith in the abilities of parents to select schools based on the numerous institutional factors that play a part in children’s educational experiences.