Yesterday, NBER released the first random-assignment study of a school choice program ever to find a negative result. Students who received a voucher through the Louisiana Scholarship Program (LSP) during the 2012–13 school year were 50 percent more likely to receive a failing score on the state math test than students who applied for but did not receive a voucher. The study also found negative effects on reading, science, and social studies tests.

The previous research on school choice had been almost unanimously positive. Out of a dozen previous random-assignment studies, 11 found positive results overall or for some subgroups, and only one found no statistically significant impact. Until now, none found any harm.

So what happened this time? As I explain at Education Next today:

Although not conclusive, there is considerable evidence that problem stemmed from poor program design. Regulations intended to guarantee quality might well have had the opposite effect. The [Louisiana Scholarship Program]’s high level of private-school regulation appears to have driven away better schools while attracting primarily lower-performing schools with declining enrollments that were desperate for more funding. 

Louisiana has one of the most highly regulated school choice programs in the nation. Private schools accepting voucher students may not use their own admissions criteria, may not charge more than the amount of the voucher, and must administer the state test. It’s no wonder then that two-thirds of Louisiana private schools do not accept voucher students. Worse, as I explained two months ago, the schools that choose to accept the vouchers along with all their attached strings are likely to be of a lower quality:

Better quality private schools that have little trouble filling their seats are less likely to accept vouchers if they decide — as two-thirds of Louisiana’s private schools did — that the regulations are too burdensome. By impeding the proper functioning of the market, regulations intended to raise quality may have the unintended consequence of lowering it.

Indeed, as Professor Jay P. Greene warned, the few schools “willing to do whatever the state tells them” to receive vouchers are those that are “most desperate for money.” According to the NBER study, “LSP schools open in both 2000 and 2012 experienced an average enrollment loss of 13 percent over this time period, while other private schools grew 3 percent on average.” The authors note that this “indicat[es] that the LSP may attract private schools struggling to maintain enrollment,” and they conclude that these results “suggest caution in the design of voucher systems aimed at expanding school choice for disadvantaged students.”

The results, said Matthew Ladner of the Foundation Excellence in Education, constitute “a very bitter lesson”:

If you are a low-income student attending low-rated schools in one of the lowest performing states, you got the short end of the stick in life. The very good people who designed this program had every intention of this program being a path out. Tragically in designing to keep bad schools out, they ironically kept the good schools out and invited the bad schools in. The road to this hell was built out of the cobblestones of good intentions, but it still led straight to this debacle.

The results are sobering but they shouldn’t be surprising. For years, factions within the school choice movement have argued over the effectiveness and wisdom of numerous regulations. For example, when the Fordham Institute released a “policy toolkit” praising Louisiana for mandating the state test, among other regulations, it launched a vigorous debate. Andrew Coulson and I argued that, while well-intentioned, uniform testing mandates would stifle diversity and innovation. Matt Ladner warned that Fordham failed to recognize the “natural limitations of technocrats” in their overconfidence about the ability of policymakers to guard against the “risk of self-defeating homogenization of the school offerings available.” Robert Enlow of the Friedman Foundation for Educational Choice cautioned that governments are “prone to politicial decisions and special interests,” and that accountability should be primarily to parents, not bureaucrats. Jay Greene noted that “testing requirements hurt choice because test results fail to capture most of the benefits produced by choice schools,” and that “highlighting a measure that severely under-states performance puts those programs in jeopardy.” AEI’s Rick Hess expressed skepticism that “policymakers have a Goldilocks-like ability to find the ‘just right’ solution” and called the Fordham report “surprisingly naïve about the realities of the legislative process and regulatory creep.”

Ultimately, the Fordhamites dismissed these concerns. Future Fordham president Michael Petrilli waved them away as mere “hypotheticals” whereas his concerns—such as parents making bad decisions and the long-term impact of crummy schools on kids—were “all playing out, right now, in the real world.” Noting that “we have to work hard to get the policy design right,” Petrilli “applaud[ed] Louisiana and Indiana policymakers for doing a darn good job on this front with their statewide voucher programs.” Then-president of Fordham Chester Finn, went further, arguing that “it’s insane to expect [the] marketplace to yield quality control, efficiency, and accountability for educational outcomes.”

(Ironically, Petrilli also expressed his fear that “bad private schools will get lots of media attention, which will drive down public support for school choice and strengthen the hand of those who opposed such programs in the first place.” Sadly, the negative findings in the NBER study stemming from Petrilli’s preferred policies are likely to do more damage to the public support for school choice than any one bad school could ever do.)

Petrilli’s concerns about crummy schools are perfectly reasonable. Indeed, all his interlocutors share them. In this debate among friends, everyone wants what is best for children. Where they differ is over the best means to improve educational quality. Given that there is no perfect system, the question is what sort of system is most likely to produce the best outcomes. 

Our friends at Fordham rested their case for test-based accountability on a single piece of evidence: according to a longitudinal study of Milwaukee’s voucher program, test scores rose in the year after the state mandated that voucher students take the high-stakes state test. However, Patrick Wolf, one of the study’s authors, cautioned Fordham against reading too much into that finding:

Ours is one study of what happened in one year for one school choice program that switched from low-stakes testing to high-stakes testing. As we point out in the report, it is entirely possible that the surge in the test scores of the voucher students was a “one-off” due to a greater focus of the voucher schools on test preparation and test-taking strategies that year. In other words, by taking the standardized testing seriously in that final year, the schools simply may have produced a truer measure of student’s actual (better) performance all along, not necessarily a signal that they actually learned a lot more in the one year under the new accountability regime.

If we had had another year to examine the trend in scores in our study we might have been able to tease out a possible test-prep bump from an effect of actually higher rates of learning due to accountability. Our research mandate ended in 2010-11, sadly, and we had to leave it there – a finding that is enticing and suggestive but hardly conclusive.

In other words, Fordham’s case rested on a rather thin reed. Indeed, with the publication of the recent NBER study, the evidence now leans strongly against the technocratic approach.

In the wake of the criticism of Fordham’s policy toolkit, Petrilli expressed an admirable willingness to change course based on the evidence:

Maybe the tests that voucher students take need not be the state tests so long as they’re solid measures of achievement. Perhaps we need to let schools point to alternative measures of student outcomes before they are kicked out of choice programs. Possibly we need an accountability regime that’s completely separate from that which governs the public schools. 

Given the latest evidence, perhaps Fordham will rethink its support for the technocratic approach altogether.