Bruce Baker, author of the study of private school spending about which I blogged yesterday, has responded to my critique. Dr. Baker thinks I should “learn to read.”
He takes special exception to my statement that he “makes no serious attempt to determine the extent of the bias [in his chosen sample of private schools], or to control for it.” Baker then points to the following one paragraph discussion in his 51 page paper that deals with sample bias, which I reproduce here in full [the corresponding table appears on a later page]:
The representativeness of the sample analyzed here can be roughly considered by comparing the pupil-teacher ratios to known national averages. For CAS and independent schools, the pupil-teacher ratio is similar between sample and national (see Figure 21, later in this report). Hebrew/Jewish day schools for which financial data were available had somewhat smaller ratios (suggesting smaller class sizes) than all Hebrew/Jewish day schools, indicating that the mean estimated expenditures for this group might be high. The differential, in the same direction, was even larger for the small group of Catholic schools for which financial data were available. For Montessori schools, however, ratios in the schools for which financial data were available were higher than for the group as a whole, suggesting that estimated mean expenditures might be low.
Even with my admittedly imperfect reading ability, I was able to navigate this paragraph. I did not consider it a serious attempt at dealing with the sample’s selection bias. I still don’t. In fact, it entirely misses the main source of bias. That bias does not stem chiefly from class size differences, it stems from the fact that religious schools need not file spending data with the IRS, and that the relatively few that do file IRS Form 990 (0.5% of Catholic schools!) have a very good reason for doing so: they’re trying harder to raise money from donors. This is not just my own analysis, but also the analysis of a knowledgeable source within Guidestar (the organization from which Baker obtained the data), whose name and contact information I will share with Dr. Baker off-line if he would like to follow-up.
Obviously, schools that are trying harder to raise non-tuition revenue are likely to… raise more non-tuition revenue. That is the 800 pound flaming pink chihuahua in the middle of this dataset. According to the NCES, 80 percent of private school students are enrolled in religious schools (see p. 7), and this sample is extremely likely to suffer upward bias on spending by that overwhelming majority of private schools. They may spend the extra money on facilities, salaries, equipment, field trips, materials, or any number of other things apart from, or in addition to, smaller classes.
Baker’s study does not address this source of bias, and so can tell us nothing reliable about religious schools, or private schools in general, either nationally or in the regions it identifies. The only thing that the study tells us with any degree of confidence is that elite independent private schools, which make up a small share of the private education marketplace, are expensive. An uncontroversial finding.
It is surprising to me that this seemingly obvious point was also missed by several other scholars whose names appear in the frontmatter of the paper. This is yet another reminder to journalists: when you get a new and interesting paper, send it to a few other experts for comment (embargoed if you like) before writing it up. Doing so will usually lead to a much more interesting, and accurate, story.
