For seven months, Secretary of Education Arne Duncan and the media have bombarded us with tales of how Duncan dramatically boosted student achievement as leader of Chicago Public Schools. Based on two new independent analyses, Duncan’s real impact appears to have been near zero.

The usual evidence presented for Duncan’s success is the rise in the pass rate of elementary and middle school students on Illinois’ own ISAT test. But state tests like the ISAT are notoriously unreliable (they tend to be corrupted by teaching to the test and subject to periodic “realignments” in which the passing grade is lowered or the test content is eased). In January, the Schools Matter blog argued that exactly such a realignment had occurred in 2006.

So to get a reliable measure of Duncan’s impact, I pulled up the 4th and 8th grade math and reading scores for Chicago on the National Assessment of Educational Progress — a test that is much less susceptible to massaging by states and districts. I then compared the score changes in Chicago to those for all students in Large Central Cities around the nation, and tested if the small differences between them were statistically significant. Not one of them is even remotely significant at even the loosest accepted measure of significance (the p < 0.1 level). Chicago students did no better than those in similar districts around the nation between 2002/2003 and 2007, a period covering virtually all of Duncan’s tenure in Chicago.

As I was finishing up this statistical analysis a few minutes ago, I came across a new report by the Civic Committee of The Commercial Club of Chicago. According to the Civic Committee report, the elementary and middle-school ISAT gains touted by Duncan and the media appear to be almost entirely illusory: artifacts of the 2006 realignment. Chicago high school students, who take a different test that was not realigned, perform no better today than they did in 2001 — so whatever real gains did occur in the early grades evaporated by the end of high school.

Writing in the Chicago Tribune a few days ago, columnist Greg Burns touted Duncan’s supposed success as CEO of Chicago Public Schools, and noted that Duncan had good prospects for winning the support of business leaders nationally, as he did in Chicago. But Chicago’s Commercial Club has now concluded that Duncan failed to accomplish what he has claimed, and given that the NAEP scores echo their findings, the education secretary may soon find national business leaders more skeptical as well.