It is understandably considered bad news for school choice when a study comes out finding negative effects on test scores, especially one using a “gold standard,” randomized control trial design. But context is crucial for understanding such findings, and this may be especially true for a new study of vouchers in Washington, D.C.

The researchers studied various impacts one year after families applied to the District of Columbia Opportunity Scholarship Program. Studied students participated in a lottery for vouchers, which is key because who wins or loses is random, automatically controlling for variables that can powerfully affect outcomes, such as family income or motivation levels.

nd those who won and used the voucher to attend a private school — what policymakers likely care most about — saw slightly lower scores than non-users. Reading scores were also slightly lower, but were not statistically significant, meaning the researchers could not be confident the differences were other than the result of random chance.

So what’s the good news here? Surely no one can be pleased that choice appears to lead to lower math scores.

For one thing, on other measures the program fared better. Parent and student satisfaction with their schools were higher for both lottery winners and winners who used their vouchers, though the results did not reach statistical significance. Both winners and users were also more likely to perceive their schools as safe, though statistical significance was only reached for parents. And for kids in sixth through 12th grade, the program had a positive impact on parents’ involvement in education-related activities at home.

Much more important, the test results may well be the result of choice working, not failing, in Washington. You see, families in Washington have lots of choices.

First, Washington is a city, so people who live there can put pressure on the district by saying, “Shape up or we’ll move to Virginia or Maryland.” More directly, Washington has a huge charter school sector. Indeed, 42 percent of the study’s control group attended charter schools, and 10 percent attended private schools despite losing the lottery. Only 48 percent attended traditional public schools.

Quite possibly because of so much choice pressure — most imposed by Congress — the public schools in Washington have seen marked increases in achievement. According to the National Assessment of Educational Progress, since the mid-1990s achievement in Washington has risen at a rate appreciably outpacing the national average. Charter schooling in Washington started in 1996, and the voucher program was created in 2004.

Alas, charter schools — tuition-free public schools that in many ways seem private — have likely hurt Washington’s private sector, which was already struggling against free traditional public schools and decades of changes in the Roman Catholic Church, whose schools predominate in the private sector. If nothing else, struggling to survive cannot be positive for staff morale.

So no one should be surprised that a voucher program enrolling fewer than 1,200 students, which has been repeatedly threatened with extinction, does not have powerful testing effects. Oh, and a maximum voucher is roughly $8,000 for grades K‑8, and $13,000 for 9–12. Meanwhile, the traditional public schools spend a whopping $30,000 per-pupil, and charters get about $17,500.

Given the gaping funding disparities, it may seem amazing that previous “gold-standard” research found that Washington voucher students performed on par with the control group on tests, and beat it soundly on high school graduation rates. And in the current report, the math difference was only about 7 percentile points between voucher users and the control group—appreciable, but not yawning.

Then there’s this: the sum of education is far more than standardized test scores. Indeed, the nation has seen a backlash against education reduced to such narrow measures, which may not predict future success. And it may be that people want things out of schooling that simply cannot be easily tested, ranging from safety, to strong moral values, to appreciation for the arts, to just a sense of fulfillment. Which is why it may be a good sign that even if it adversely affected test scores, schools chosen by lottery winners spent less time on math and reading instruction than control group schools. They may instead have been devoting time to music, or field trips, or myriad other very valuable activities.

Are negative testing impacts good? All other things equal, no. But all other things are not equal, especially in Washington.