In recent weeks, the Fordham Institute has repeatedly called for government testing and reporting mandates to be imposed on private schools participating in school choice programs (here and here), on the grounds that such “public accountability” improves private school academic outcomes. In defense of this claim, the Fordham Institute cites a study of Milwaukee’s voucher program in which test scores rose following the introduction of such mandates.

Patrick Wolf, director of the research team that conducted the study, has now responded, explaining that his team’s results do not necessarily support Fordham’s claim:

[B]y taking the standardized testing seriously in that final year, the schools simply may have produced a truer measure of student’s actual (better) performance all along, not necessarily a signal that they actually learned a lot more in the one year under the new accountability regime….

What about the encouraging trend that lower-performing schools in the MPCP are being closed down? [Fordham] mentions that as well and attributes it to the stricter accountability regulations on the program. That phenomenon of Schumpeterian “creative destruction” pre-dated the accountability changes in the choice program, however, and appears to have been caused mainly by low enrollments in low-performing choice schools, as parents “voted with their feet” against such institutional failure. Sure, the new high-stakes testing and public reporting requirements might accelerate the creative destruction of low-performing choice schools in Milwaukee, but that remains to be seen. [emphasis added]

But there is a deeper problem with the Fordham claim, to which Wolf alludes: a single study, no matter how carefully executed, is not a scientific basis for policy. Because a single study is not science. Science is a process of making and testing falsifiable predictions. It is about patterns of evidence. Bodies of evidence. Fordham offers only a toe.

And Fordham’s preferred policy not only lacks a body of supporting evidence, it is undermined by a large body of evidence. When I reviewed the within-country studies comparing outcomes among different types of school systems worldwide in 2009, I sorted the results into two categories: 1) all studies that compared “public” schools to “private” schools, where those terms were loosely defined; and 2) studies that compared “market” schools to “monopoly” schools. “Market” schools were those paid for at least in part directly by parents and only minimally regulated. “Monopoly” schools were public school systems such as those common in the U.S.

The purpose of these separate categorizations was to see if limited regulation and direct parent funding make a real difference, or if private schools that are paid for entirely by the state and subjected to Fordham’s “public accountability” have the same advantages as their more market-like counterparts.

The result of this breakdown of the literature was stark. Studies looking at truly market-like education systems are twice as consistent in finding a private sector advantage as those looking at “private” schools more broadly construed (and thus including state-funded and regulated private schools).

The pattern of evidence thus seems to contradict Fordham’s belief in the merits of “public accountability” in market education systems. What it favors are policies that promote the rise of minimally regulated education markets in which parents pay at least some of the cost of their own children’s education directly themselves, whenever possible. That’s just the sort of system likely to arise under education tax credit programs.