Both the Wall Street Journal and the New York Times jumped on the release of a new study by the National Center for Education Statistics this weekend. The WSJ’s headline was particularly dramatic: “Long-Delayed Education Study Casts Doubt on Value of Vouchers.”


No, it doesn’t.


And it is a failure on my part, as well as a failure of the school choice movement as a whole, that the media don’t understand why.


Taking the study entirely at face value, what it says is this: private school students consistently score better in math and reading on the National Assessment of Educational Progress (NAEP) than public school students, but their advantage essentially goes away if you apply a particular set of controls for the differing student characteristics between the two sectors (things such as wealth, race, etc.)


Okay, you say, but if private schools don’t significantly outscore public schools, what’s the point of school voucher programs or other reforms that would give all parents access to the public or private school of their choice? Why, in other words, is the Journal’s headline wrong?


It’s wrong because the point of voucher programs is to create a competitive education industry, and the existing population of U.S. private schools does not constitute such an industry.


A vigorous free market in education requires that all families have easy access to the schools of their choice (whether public or private); that schools are not burdened with extensive regulations on what they can teach, whom they can hire, and what they can charge, etc.; that consumers directly pay at least some of the cost of the service; that private schools not be discriminated against financially by the state in the distribution of education funding, and that at least a substantial minority of private schools be operated for profit.


This set of conditions does not exist in any state in the nation. Instead, American education is dominated by a 90 percent government monopoly that is funded entirely through taxation. The private sector occupies the remaining 10 percent niche, is almost exclusively operated on a non-profit basis, and is forced to charge thousands of dollars in tuition in the face of the “free” monopoly schools that spend an average of $10,000 per pupil per year.


This is not a market.


No study was necessary to point this out.


Competitive markets are characterized by innovation, inexorable improvements in cost effectiveness and the quality of goods and services, and the rapid growth of the most successful providers. None of this has occurred in the U.S. private education sector, precisely because that sector does not constitute a competitive market.


The last great innovation to transform classroom instruction occurred during the presidency of Thomas Jefferson: the invention of the chalkboard, around 1801. Since that time, the pace of innovation has been so slow that a student from the mid-1800’s would immediately recognize a modern classroom setting. The most sought-after private schools enroll only about a thousand more students today than they did a century ago. This degree of stagnation is unheard of outside of the education sector, because it is only in the education sector (at least in liberal democracies) that market activity has been so thoroughly extinguished by government monopoly provision.


Hence, this study of our current small, non-market niche of private schools does not allow any generalization to the sort of outcomes to be expected from a true free market in education—and the creation of such a market is the primary justification for voucher and other school choice policies.


If I were better at my job, and if the school choice movement as a whole had a more effective media machine, this fact would be widely understood and we wouldn’t see fallacious headlines like the one cited above.


A closer look at the findings


That major point having been made, let’s take a look at the study’s findings on their own merits, as an examination of the current crop of public versus private schools.


The first problem with the study is that it collects no data on per-pupil spending in public versus private schools. Private school tuition, according to the NCES itself, is about half of the average public school expenditure per pupil. While private schools have some other sources of revenue, they still spend thousands of dollars less per pupil than public schools even after taking these other revenues into account, and so may be dramatically more efficient even if their absolute achievement levels are comparable to those in public schools. Hence it is possible that, if spending were equalized, private schools would raise student learning substantially compared to current levels (while it has been shown that spending and achievement are largely unrelated in the public sector, this has not been demonstrated in the private sector. In fact, evidence from developing countries suggests that higher spending in private schools DOES increase student achievement).


Next, it is worth observing the specifics of the study’s findings. It reports that there is a small advantage to public schools in 4th grade math, but that this advantage is not present at the 8th grade. It further says that, at the 8th grade, private school students have a small advantage over public school students in reading. One possible interpretation of these findings is that public school students fall behind their peers in private schools the longer they spend in the classroom.


That, of course, is only one possibility. At any rate, it is clear that parents are most concerned with what their children know and are able to do at the end of their k‑12 education, so if, by the later grades, private schools confer a significant advantage, this would definitely seem to favor them.


Methodological and Data Problems


All of the above discussion takes the study’s findings at face value. This may be ill-advised, since a preliminary review suggests that there may be real methodological problems and potentially serious data problems. Several of the control variables used in the model seem problematic, including the following.

The rate of student absenteeism

It is entirely possible that sectoral differences in the feeling of community or level of personal attention, ability of school staff to motivate students, etc., could affect student absenteeism. So it is erroneous to treat this as exogenous (i.e., as independent of school sector) and to control for it.

School size

This variable is clearly endogenous (i.e., affected by school sector). Parents tend to prefer schools in which teachers know all the students by name and which create a friendly, community atmosphere. This is much easier in smaller schools, and hence there is a competitive pressure not to get too large in the private education sector. No such pressure exists in the public sector, where contrary bureaucratic incentives encourage large school size. As a result, the average public school is roughly three times the size of the average private school: 521 students versus 182. It is thus unjustifiable to pretend that school size is independent of school sector.

The percentage of students in the Title I program

A report by the Congressional Budget Office notes that “About 97 percent of public schools and 45 percent of private schools participate in the school lunch program.” This vast difference in level of participation by schools may have a significant effect on the share of eligible students who are in fact being served by the program.

Sample specification problems

Between a fifth and a quarter of the private schools selected for the study did not participate. The authors make no serious attempt to analyze non-participants to determine how and to what extent they might differ from participating schools in ways related to student performance. This could bias their results in unknown ways.


It seems likely that public and private sector schools apply the federal Specific Learning Disability label differentially. This label states that children are disabled if they perform at a level below what would be expected for students of their age and intelligence. It does not account for the possibility that poor performance may be the result of poor instruction. Roughly six percent of all public school students are placed in this category, making up nearly half (43%) of all students classified as disabled in the public sector. Among private schools participating in this study, a total of 3 to 4 percent of students are classified as suffering from ANY disability, mental or physical. Because students classified as SLD can be excluded from the test taking pool or given extra time or other accommodations, differential SLD classification rates between the sectors may affect sectoral mean scores (because these students, by definition, perform below the average of their peers).

Instrument selection

Tom Loveless has pointed out in a paper for the Brookings Institution that the NAEP mathematics test does a poor job of measuring the skills that it is purported to measure. Calculator use is allowed throughout, so it does not measure basic arithmetic ability. More advanced topics such as algebra with fractions, are also all but absent, making it a poor test of these more advanced skills. If there are differences in either of these important areas between the sectors, the NAEP will not pick it up. It is natural for scholars to want to analyze the data they have, but readers should be aware of the shortcomings of those data as a measure of both basic and advanced mathematical ability.

Conclusion

Taking all of the above analysis together, this study’s findings would have little bearing on market-based reforms such as vouchers and tax credits even if it were methodologically flawless. Even as a comparison of public schools and the existing (non-market) crop of private schools, it leaves much to be desired because it neglects to consider the substantially higher per-pupil spending of public schools.


But the study, as noted above, is not methodologically flawless. Several of its control variables appear to be misspecified, and so its adjusted test score averages may be significantly biased. It makes no attempt to assess the impact of the non-participation by between a fifth and a quarter of all the private schools selected for participation in the study – another probable source of bias. And it uses a mathematics test (the NAEP) that has been shown to do a poor job of assessing both basic arithmetic and more advanced mathematical skills, thus obfuscating possible differences in performance in these (rather important) areas among the students tested.


In a nutshell: this study does not say what some reporters think it says, and it may not even say what its own authors think it says.