The Trouble With the Challenge Index

Sara Mead demolishes Jay Matthews' "Challenge Index".

I'm thrilled to see Sara on The Washington Post's op-ed page, but it's worth saying something here about the cynicism. Matthews is an education reporter for the Post and for Newsweek (which published the index) and which are both part of the same company. They've been publishing his list for years. And Sara and her now-former boss Andrew Rotherham have been offering their criticisms for quite some time.

In short, this isn't some new controversy that just dawned upon the relevant editors, who ought to consider trying to make up their minds. If the Mead/Rotherham critique is correct, they should stop publishing the index. If the Mead/Rotherham critique is wrong, they should stop publishing the critique. Meanwhile, Sara and Andy are both far too polite to point out that the same corporation that insists on ranking high schools based purely on the number of AP and IB tests their students take also happens to own the country's major purvey of standardized test preparation services.

Comments

I'd say that the demand for test preparation services is independent of Jay Matthews's particular method of measuring high schools.

Posted by otto | June 24, 2007 12:40 PM

"Sara, one of the most talented education policy players in the country"

Ugh - Rotherham's choice of words suggests the essential frivolity of DC-for-twentysomethings. (Not disputing her talents, etc, but this sort of expression makes politics sound like a lifestyle choice more than anything else).

Posted by otto | June 24, 2007 12:47 PM

I've always wondered why, if college-lite programs like AP or IB say so much about a school's worth, college courses taken in high school are not considered in Newsweek's formula. For example, my high school doesn't offer any AP or IB courses, but it does allow students to take classes at Dartmouth; under Newsweek's formula, this counts for nothing. But no, this couldn't have anything to do with the Washington Post owning Kaplan, certainly not.

Posted by Minipundit | June 24, 2007 1:12 PM

Further to otto, the implication that Jay Mathews -- the WP reporter who devised the Challenge Index -- has been paid off in some way for the way he designed it is both stupid and false. If you don't like the Challenge Index, that's fine, but the innuendo is just silly. (In particular, I'm not aware that anyone has devoted as many column inches to criticisms of the Challenge Index as the Washington Post.)

On the substance, Jay Mathews has always admitted that his Index is flawed (or at least limited in its scope), and his response has always been that if you can come up with a more meaningful Index based on statistics about schools that are actually available in the real world, knock yourself out. An op-ed contending that someone ought to do so does not count to my mind as a demolition.

Posted by alkali | June 24, 2007 1:31 PM

Jay Matthews' challenge index has no external validity, but pushes schools to extend AP courses.

I've had intersting email exchanges with Mr. Matthews and I think he is well meaning. He is, for example, well aware of elite schools, such as Scarsdale High School (where his kids went to school) that are attempting to do away with AP courses. These schools obviously suffer under Mr. Matthews' ranking system.

In actuality, Mr. Matthews has an agenda. He believes that challenging underperforming kids with difficult material is the best way to improve the high school system. His book, "Escalante: the best teacher in America" (you may remember the movie based on this story, "Stand and Deliver") espouses that same notion. This is an interesting idea, perhaps a good one, but unproven. To use this ranking system to further this notion is wrong.

Let me give a personal example. I live in a community in NJ. New Jersey Magazine, copying the Newsweek method, has ranked NJ high schools based on the AP criteria. Our high school's rating recently dropped. At two PTA meetings I recently attended parents brought this up and challenged the superintendent to improve the scores. (One of the stated goals was to maintain real estate values; educational goals are easily trumped). The superintendent, on the defensive, suggested difficulties with the rating system and and began to discuss mechanisms for expanding AP participation. The mechanisms suggested involved paying for students to take the AP tests (they are expensive) and expanding AP offerings. If this is done it will be at the expense of offering more a diverse course selection and other forms of innovation. Mr. Matthews is winning. And, as the post article points out, there is little empirical data suggesting that his formula has real benefits.

Finally, I'd like to point out that, by and large, colleges do not like the AP system. Fewer and fewer colleges give credits for AP courses.

Personally, I feel that American High Schools are largely broken, and that the AP solution is wrong-headed. I'm inherently suspicious of any ranking system. When ranks are based on terrible measurement tools, problems will follow.

Posted by John Kubie | June 24, 2007 2:01 PM

It sounds like the "Challenge Index" is of little value in evaluating the overall performance of a school.

However, it may have great value for an upper-middle class urban parent to evaluate how good a school is likely to be for her particular child.

So it's not useless, it just is what it is.

Posted by ed | June 24, 2007 4:41 PM

Response to Ed.

Actually, Matthews wants to have the challenge index change weak schools, as in the "Stand and Deliver" movie. Upper middle class kids in upper middle class suburbs go to high achieving schools, some of whom haver rejected AP courses and exams. There is a wave of this type of rejection in private schools and it is beginning to happen in public schools, such as Scarsdale High School.

Posted by John Kubie | June 24, 2007 5:31 PM

second response to Ed:

In one sense, the "Index" is just what it is. But Matthews has a powerful position and has abused it. If on my education blog I choose to make some silly index, that's fine. But the Newsweek index is the only index I'm aware of for ranking ALL US high schools. In addition, the ranking has the stamp-of-approval of two major news organizations. As I tried to note above, it has sufficient weight to affect educational at local levels. People do not say, "you may find this index useful for selecting a high school that emphasized AP exams" (which is what it does) they say "I want my go to the high school with the best ranking". Or "this high school is (good/bad) because it has (improved/not improved) its rank".

There is very little nuance. Neither the washington post nor newsweek sell it as a nuanced ranking useful for a small number of students.

Posted by John Kubie | June 24, 2007 5:41 PM

This is just another example of the complete lack of realism that's pervasive on the topic of education in America. Practically nobody in a position to influence educational policy knows how to reason statistically.

Posted by Steve Sailer | June 24, 2007 5:50 PM

Mathews is another jaded '60s liberal depressed by the fact that the countercultural revolution has failed. Specifically, it depresses him that 50 years of lefty educators in charge haven't figured out how to raise the test scores and general scholastic achievement of black kids significantly. He lives to close to educational ethnic gap.

His "Challenge Index" comes in for a lot of criticism because it's incredibly stupid. You don't rank high school basketball teams on how many practices they hold, you rank them on how many games they win.

Mathews' Index is just another educrat stat designed to fool the masses (and the dwindling believers among educrats) that progress is not only possible, but just around the corner.

Mathews' other trick here is to cherry-pick selected edustats among normal fluctuations and present them as progress. Typical for him is something like "SAT scores of African-American students in New York State jumped 11 points in 2005, a promising yada yada."

He doesn't tell you they dropped 8 points the year before and haven't moved overall in 40 years.

Providing hope that the ethnic gap will disappear in the future is the Challenge Index's sole raison d'etre, and the psychological driver for Mathews creating and promoting something so obviously pointless.

Posted by Zagnut | June 24, 2007 8:08 PM

In short, this isn't some new controversy that just dawned upon the relevant editors, who ought to consider trying to make up their minds. If the Mead/Rotherham critique is correct, they should stop publishing the index. If the Mead/Rotherham critique is wrong, they should stop publishing the critique.

Come on, Matthew. As Mark Jackson would say, you're better than that.

Howabout, the editors feel that the index has some things going for it, as does the critique. The critique is neither completely correct nor copletely wrong. They therefore believe that presenting both sides of the issue allows their readers to make that determination for themselves.

As to the substance of Sara's argument, Matthews deals with it in his article. He recognizes that "best" means different things to different people. Perhaps some people will agree with him about the value of what's reflected in his index, and will disagree with Sara about the value of looking at racial gaps, etc. Why not give people that option?

Posted by Al | June 24, 2007 9:58 PM

I agree with Al. The Post is trying to give its readership diverse perspectives on the issue school-ranking. Not everybody thinks a blogospheric echo chamber with total editorial consistency has all the answers.
As for the merits of the debate, I will soon have insightful commentary on my blog, the greatest in the world that will answer all of your questions on this subject.

Posted by Wido Incongitus | June 25, 2007 4:22 AM

I think Matt is mostly right.
The prior two posts suggest that Jay Matthews' column is something akin to an op-ed, and that publishing another op-ed is balance. But the Matthews piece is far from an op-ed. It is published by Newsweek in great detail and by the washington post in summarized form as an original and major piece of research that has fundamental impact on educational policies. This is treated as raw data, that is used by many in making decisions. It certainly has the imprimatur of Newsweek and, by extension, the Washington Post. These publications do not fund similar research by competing "scholars". Importantly, it is nothing like an op-ed. Unfortunately, I don't really understand Matt's argument. The two types of articles are not symmetric and don't balance. I think the argument Matt should make is that the Matthews work is bad scholarship and bad policy. Furthermore, given his prominent position, it has undue influence. For a variety of reasons, only some of which are mentioned in the Mead/Rotherham critique, it should not be funded and published.

So, what about the money angle? I think Matt is right to smell money, but he smells it in the wrong place. There are two likely aspects to the money angle. First, this sort of ranking sells papers and magazines. Newsweek is copying other news magazines in this game. All sorts of things are ranked, from football teams to pizza shops. It sells. Many of these are trivial and fun. But many have really bad effects. In this case, the effects are bad. A second place to look for a money influence is the college board corporation, the company that makes money on administering AP tests and other exams. I doubt that Newsweek and the Washington Post have direct interest in College Board, but it is certain that they like Mr. Matthews' work.

Posted by J | June 25, 2007 10:51 AM