In the author’s note to Smartest Kids in the World, Amanda Ripley writes: “I didn’t care deeply about charter schools, vouchers, tenure, or other policy hang-ups…. So, I thought, I’ll just slip out the back door and go investigate this other mystery for a while.” That other mystery was the apparent ability of some countries to educate their children unusually well.
Ripley’s note captures both the book’s strengths and its weaknesses. She is a talented writer with a sense of adventure, and her prose is a pleasure to read. By setting aside the leading education policy questions of our time, she is able to focus on telling the personal stories of children from very different parts of the world, and there is much to be learned from them.
But there is a cost to ignoring virtually all of the evidence on how education policy affects educational outcomes: you’re much less likely to find the needle in the haystack if you decide not to look at the hay. When Ripley concludes that the effect of policy is marginal, the reader can only wonder: how would she know, when she didn’t study the evidence?
Even in selecting her “smartest” countries, Ripley seems not to have done her homework. Her sole performance metric is PISA—the Program on International Student Assessment. PISA tests 15-year-olds on their ability to apply basic knowledge of math, science, and language to everyday problems. Ripley, who took the test, described it as “more like a test of life skills than school skills.” Typical questions might include interpreting a picture of a gas-gauge or a simple bar chart.
It’s interesting to know how well 15-year-olds in various countries perform on PISA, but it’s not clear why anyone would use it as the exclusive metric for identifying educational quality. There are other outcomes worth measuring.
Even within the realm of international academic tests, PISA is not alone. The Trends in International Mathematics and Science Study (TIMSS) focuses less on word problems and more on the subject-specific knowledge and skills that schools teach. It includes more technical questions that measure how well-prepared students are for further study in those subjects.
Critics of TIMSS argue that it doesn’t test whether or not students can apply their mathematical and scientific knowledge to everyday problems, while critics of PISA allege that it doesn’t measure the skills needed to succeed in college or technical professions. Since these perceived deficits are complementary, examining the results of both tests seems more useful than looking at just one.
Ripley’s exclusive use of PISA led her to choose Korea, Finland and Poland as the homelands of the best-educated kids in the world. Poland is chiefly included because its PISA performance improved substantially in the past decade, placing it on par with the U.S., despite its vastly lower level of economic development. Korea and Finland make the cut because they are high-flyers. Korea, in fact, is a top performer on both the PISA and the TIMSS, scoring far above the international average in every subject and grade. But the picture for Finland is quite different. It performs comparably to Korea on the PISA test, but lags well behind it on the TIMSS.
In 4th grade math, Korea is essentially tied with Singapore for the number one spot on TIMSS, scoring an amazing 605 versus an international average of 500. Finland’s score of 545 is far lower—statistically indistinguishable, in fact, from that of the United States (which scored 541). In 8th grade math, Korea and Singapore are again at the top, scoring 613 and 611, respectively, versus the international average of 500. At the same grade, Finland scores only 514—statistically indistinguishable from the 509 earned by the United States and a staggering 99 points below Korea. Moreover, Finland’s scores on the TIMSS declined significantly over the past decade.
As the Brookings Institution’s Tom Loveless has observed, Finnish mathematics professors have noticed a discrepancy between their country’s high PISA math scores and their students’ actual ability to do math. In 2005, two-hundred of them signed a petition bewailing the mathematical under-preparedness of their country’s college and technical-school freshmen—consistent with its middling and declining TIMSS performance.
Though Ripley neglects to discuss the limits of her PISA litmus test in light of TIMSS scores or the concerns of Finnish math professors, there are some areas in which the book cites and comports with the relevant scientific research. It correctly notes, for instance, the lack of any consistent relationship between levels of per pupil spending on public school systems and the performance of their students.
There are other cases, though, which are more problematic. In a section titled “Gifted and Talented in America,” Ripley states that “tracking tended to diminish learning and boost inequality wherever it was tried.” This, according to the endnotes, is a reference to a comparison of PISA results between countries that permanently assign students to different academic tracks as early as age 10 (often to entirely different schools) versus countries that introduce permanent tracking no earlier than age 16, if ever (Hanushek and Woessmann, 2005).
Despite the narrow focus of that study, Ripley goes on to make broad claims and implications about student grouping practices, without examining what the literature has to say about them. For instance, Ripley writes that kids placed in gifted classes “at age eight probably tended to see themselves that way.” As it happens, this claim is precisely backwards. When Kulik and Kulik reviewed the self-esteem effects of grouping students by ability they found that children placed in higher-performing classes came to think of themselves as less gifted while those placed in lower-level classes came to think of themselves more positively. When surrounded by children who achieved at a level similar to their own, whatever that level happened to be, students came to realize that they were not so special after all. Lower achievers felt less stigmatized, while higher achievers were sobered by the presence of similarly capable peers.
Nor does Ripley seem aware of the distinction that researchers have found between the permanent, across-the-board tracking studied by Hanushek and Woessman and flexible grouping and re-grouping of students based on their performance in particular subjects, at particular times. In the midst of her negative discussion of tracking, Ripley seems incredulous that the United States not only “divided younger children by ability, but actually taught different content to the more advanced track” [italics in the original]. In reality, targeting the content and rate of instruction to students grouped by performance shows some of the most powerful positive effects in the education literature. In a systematic review of that literature, Kulik and Kulik found that
Programs of enrichment and acceleration, which usually involve the greatest amount of curricular adjustment, have the largest effects on student learning. In typical evaluation studies, talented students from accelerated classes outperform nonaccelerates of the same age and IQ by almost one full year on achievement tests.
This only stands to reason. Children allowed to progress through the curriculum at their own pace learn more than those compelled to continue studying topics they have already mastered.
Even when the curricular adjustments are more modest, students in flexible, performance-based classes, “outperform equivalent control students from mixed-ability classes by two to three months on a grade-equivalent scale.” And this applies to students in every performance group, not just to the highest performers.
So while Ripley is justified in scorning the “ghetto effect” of permanently tracking students across all subjects, she misleads her readers by overgeneralizing and by failing to discuss the substantial benefits of flexible performance-based grouping.
In short, the Smartest Kids in the World, is not the sort of systematic and thorough work that could identify the students referred to in its main title, or that could explain their performance as promised by its subtitle. It is a work of journalism, and it suffers the same problem that journalism usually faces when grappling with vast, complex subjects: the author cannot become an expert in the time available. That is why the great works popularizing such topics are generally written by experts who have a gift for communication—the Carl Sagans of the world—rather than by journalists.
Despite these caveats, I thoroughly enjoyed the book and recommend it to those who study education policy. It offers animated and interesting stories about the lives of students with widely varying educational experiences. Those stories have value, and they are well told. Readers who work in this field will be able to pick out the book’s errors or at least easily double-check questionable claims.
General readers may come away misinformed on a variety of topics, however, unless they focus on the book’s personal stories and treat the rest with a grain of salt.
