Demand-Side Solutions

An interesting article details efforts by some college administrators to sabotage the US News and World Report rankings by getting enough schools to agree to decline to provide the information they're asked for. It's a good idea. TheUS News rankings are a terrible farce and killing them off would be a good thing. This even seems like a reasonable tactic.

All that said, the very best way to deal a death-blow to this scheme would be for America's colleges and universities to work together and with third parties to try to come up with some meaningful metrics for higher education performance. All magazines make lists, but the reason the college rankings are such a hit is that there's nothing out there. Ordinal rankings are inherently kind of dumb, but higher education leaders both can and should come up with some kind of theory about what service they're providing to students and some method of measuring how well they're doing it. Since the schools don't do anything like this themselves, and since their lobbyists are wildly opposed to having the government do it, the upshot has been to outsource the function to a struggling newsmagazine that deploys screwy formulae to boost sales.

If the higher education community itself provided some kind of better product, then university presidents wouldn't find themselves under US News' thumb.

Comments

higher education leaders both can and should come up with some kind of theory about what service they're providing to students and some method of measuring how well they're doing it.

I can't help thinking that this is just too facile a description of what's needed. By analogy, I might ask MY to come up with a theory about the service that political news magazines (or perhaps even blogs) provide their readers and some method of measuring how well they're doing it. It's easy to think of shallow measures, such as readership numbers; who buys this?

Posted by RSA | May 20, 2007 5:54 PM

How about: do you employ Stephen Pinker? If so, subtract 1000 from your score.

Posted by david | May 20, 2007 6:05 PM

I can't help thinking that this is just too facile a description of what's needed. By analogy, I might ask MY to come up with a theory about the service that political news magazines (or perhaps even blogs) provide their readers and some method of measuring how well they're doing it. It's easy to think of shallow measures, such as readership numbers; who buys this?

College is a much more important investment; people spend well in excess of $100,000 on an education, and they have a right, I think, to want and expect more information about what they're buying. College, for some people anyway, becomes a huge source of pride/shame/identity. In other words, people invest too much--both psychologically and financially--to not want clear markers and information about what they're getting.

I also don't see what could be so bad about groups of colleges--the NESCAC or Ivy League, say--developing mission statements and then metrics to help figure out if they're doing what they aim to do...

Right now, US News rankings are mostly about metrics that have nothing to do with being educated--endowment size, reputation, percentage of alumni who donate--and everything to do with what sort of social/cultural/professional advantage will you gain by going to a given school. If universities are providing a different service, and I'd like to think that they are, what's wrong with asking educational leaders to find some way of describing what they provide and disseminating it?

Posted by Brad | May 20, 2007 6:18 PM

My dad was a professor at Wesleyan University for over 30 years. I grew up on campus and still stay in touch with many on the faculty.

Anyway Wesleyan is firmly ensconced in the second tier, I guess; but while they are very well regarded, they don't have the name appeal of an Amherst or Williams. So a few years ago I was talking to a professor and he was telling me about how much angina the administration had because Wesleyan had been number eleven in the US News list for a couple years running, and they were desperate to become a top ten school. So they did this investigation, and found that US News put a very high premium on a senior exit survey. And the administration realized that they had been consistently distributing the survey early in the morning, the day following the biggest party night of the year. So they started giving it out during reading week, in the afternoon, when no one was tired, hung over or stressed from work. The next year they placed ninth.

What a fucking farce.

Posted by Freddie | May 20, 2007 6:22 PM

I'm sure you met plenty of people back in your Cambridge days who'd love to see the law school rankings go. Ba-zing!

That said, I think that a big part of the problem is that one of the main impediments to developing actual theories of educational value is the tension between the educational mission itself and schools' market status as credentialing shops. The idea, I take it, is if you blow up the USNWR rankings, the market for credential value will become less fierce and there'll be less institutional inertia against pursuing the education-value model.

Posted by aeroman | May 20, 2007 6:22 PM

So BMM if you think the USNWR rankings are so bad why don't you and the Atlantic come up with a better set? Of course the truth is college administrators don't want to be evaluated by anybody regardless of the criteria used.

Posted by James B. Shearer | May 20, 2007 6:36 PM

That said, I think that a big part of the problem is that one of the main impediments to developing actual theories of educational value is the tension between the educational mission itself and schools' market status as credentialing shops.

I don't remember who said it or how it was proposed, but someone once asked his students something along the lines of "If you had to choose, would you keep the Ivy-league education without the name, or the name without the education?" (Sorry for a paraphrase of a quote from a dimly remembered anecdote.)

Because, pragmatically, most people have to choose the name. I mean, if you go to Harvard, you are expected to receive an excellent education. Not the best undergrad, most would say, but an absolute top-notch education. Leaving aside the question of whether that's true-- it's the assumption that have received that great education that has financial value, not the education itself. When you go to get a job, the name on the transcript matters far more than the actually education the transcript describes.

Posted by Freddie | May 20, 2007 6:37 PM

Fair enough questions, but I'm finding the post and most comments vague, and I'd love to hear MY articulate more what kind of ranking he'd like. For all the US News ranking's problems, I think, say, that the Washington Monthly is far worse, by having even less interest in things I most wish from a university: promoting free inquiry and getting students a great education. (Not that these aren't complimentary in some regards; it's too simple to see research as antithetical to teaching, much as recent coverage of the Ivies and the "second tier" in The NY Times is making it a truism.)

Instead the Monthly stresses an idea of public service that boils down to militarism and careerism: ROTC and law/business schools. The liberal-arts and other universities that rank low with this should be our exemplars.

Posted by John Haber | May 20, 2007 6:50 PM

I'd like to see someone propose a set of "metrics" for valuing a liberal arts education at any institution, not just one in the Ivy League. Most of the subjects of such an education aren't really measurable in the manner of evaluating sales efforts, for example.

Posted by JackD | May 20, 2007 6:56 PM

For what it's worth, I'm not saying that, for universities, some kind of theory about what service they're providing to students and some method of measuring how well they're doing it wouldn't be a valuable thing. I'm saying that universities have been around for a thousand years, give or take, and that theories of education have been around for about a hundred years, give or take. Saying that theories and metrics would be nice isn't saying anything new. I don't think anyone disagrees. The state of the art just isn't there yet.

On this: I also don't see what could be so bad about groups of colleges--the NESCAC or Ivy League, say--developing mission statements and then metrics to help figure out if they're doing what they aim to do.

Mission statements at the university, college, and department level are more the rule than the exception. And talk to faculty members in pretty much any discipline about university administration: everyone seems to be interested in metrics. A lack of good metrics isn't for lack of trying.

Posted by RSA | May 20, 2007 7:10 PM

I'd like to see someone propose a set of "metrics" for valuing a liberal arts education at any institution, not just one in the Ivy League. Most of the subjects of such an education aren't really measurable in the manner of evaluating sales efforts, for example.

Right. There's are things that can't be quantified in life; I don't think there's any way to accurately measure the quality of a college or university. The number of variables are too numerous, too idiosyncratic and too inconsistent to ever judge in a meaningful way.

Posted by Freddie | May 20, 2007 7:13 PM

Median alumni income! First you'd have to studies to see how much you'd have to offer a sample to get similar rates of participation at different income level, but then you just send out forms to alumni from the class graduating 10, 20, 30 years earlier in which you ask them to release their tax records and that you'll pay them different amounts of money depending on how much money they earned in the past year. Of course potential college student has his or her own criteria in picking a college, but this would probably be an important one.

Posted by Trolly McTroll | May 20, 2007 7:15 PM

Sorry for the bad syntax and style in my last post.

Posted by Trolly McTroll | May 20, 2007 7:17 PM

There are things that can't be quantified in life

The owner of e-harmony would probably disagree, but I hate that smarmy git anyway.

Posted by RSA | May 20, 2007 7:19 PM

The most refreshingly direct rankings are those of business schools, which weigh heavily the employment prospects and starting salaries of their grads.

Still, the problem with ranking b-schools is largely the same problem with ranking undergraduate schools: the economic value and status value of a degree from any school is determined by how exclusive that school is, not by how well the school educates its students. Thus, there may be small liberal arts schools that do a better job of educating their students than Harvard does, but an employer would still rather hire a Harvard grad, because, chances are, the Harvard grad is a whole lot smarter and more capable than the little liberal arts school grad. Why? Because the Harvard grad had to be super-smart and capable to get into Harvard.

If more IQ-like tests were legalized in employment, than the value of going to expensive Ivies would be diminished, since smart, frugal kids could go to 'State and still demonstrate their intellectual chops.

Posted by Fred | May 20, 2007 7:26 PM

Mission statements at the university, college, and department level are more the rule than the exception.

Fair point. Incidentally, I'm in the process of finishing an MA and, yes, it's true, universities have mission statements and the like.

On the other hand, an associate chancellor here recently sent out a campus wide email asking for advice on how to market and brand our school. If the mission here were really education, and only education, we wouldn't get email like that.

I'm also not sure that the lack of good metrics "isn't for lack of trying." The reward for doing research at universities is the opportunity to not teach. Get a grant publish a book, and you can spend the semester in your office avoiding all those simple, pesky students. If universities really focused on education,
this would never be the case.

In that sense, I'll offer this as a starting metric: Percentage of undergrad classes taught by tenured faculty. Faculty reputation really only matters, after all, if they actually interact with students.

Posted by Brad | May 20, 2007 7:48 PM

Some guy came up with his own rankings for MFA in writing programs which is far closer to representing the interests of people who attend these programs than the US News rankings (which are based largely on some nebulous idea of reputation and don't factor in things like funding; putting aside the question of what kind of moron would drop 60k on a creative writing program there's the larger question of what kind of asshole would promote a 60k creative writing program). I don't agree with every pick but this guy's list (granted there is nothing one can do about the location of say Texas but who the hell wants to live in Texas even for like two years?) is a lot less absurd than the US News one.

Posted by Linus | May 20, 2007 7:55 PM

Median alumni income! First you'd have to studies to see how much you'd have to offer a sample to get similar rates of participation at different income level, but then you just send out forms to alumni from the class graduating 10, 20, 30 years earlier in which you ask them to release their tax records and that you'll pay them different amounts of money depending on how much money they earned in the past year.

Excellent idea, though at least for the first ten years you'd have to take the graduates' economic backgrounds into account. Universities that draw heavily from affluent families are going to have a head's-up over those with more middle- or working-class student bodies. Once you get past ten years since graduation, however, I'd expect this factor to become less significant.

Geographical adjustments also would be necessary. Universities located in, for example, Rust Belt states would be at a disadvantage compared to Sun Belt counterparts. Of course this is mitigated somewhat by graduates moving to other parts of the country, but you probably can't ignore it.

These things being said, I agree that alumni income is very, very important.

Posted by Peter | May 20, 2007 7:56 PM

As a future elitist who went to an Ivy League university somewhat worse than Harvard but somewhat better than Brown, I'd say there's some value to US News-type rankings, though not in their current, numerical iterations.

There are indeed easy ways for universities to game the system, in ways that benefit no one but the college itself. That said, there is a value in a third-party informing students--and the future employers or grad school educators of these students--about the relative quality of their undergraduate education. Ranking schools numerically is silly (is MIT four ranking spots better than CalTech, or five?) but placing schools in quality tiers would do a large amount of good without too much of the ankle-biting and ratings-grubbing that comes with the numerical ranking system. In other words, it's valuable for students and other parties to know that NYU is not as good a school as Columbia, but not that it's 24 ranking spots worse, or what have you.

Posted by Ben | May 20, 2007 8:02 PM

Ranking schools numerically is silly (is MIT four ranking spots better than CalTech, or five?) but placing schools in quality tiers would do a large amount of good without too much of the ankle-biting and ratings-grubbing that comes with the numerical ranking system.

True, but your left with a similar pickle: what about the school that is the best in tier three and the one that is worst in tier two? You've still got largely arbitrary distinctions being made, and your still going to eliminate the kind of nuance that you would need to make this kind of thing truly useful.

In other words, it's valuable for students and other parties to know that NYU is not as good a school as Columbia, but not that it's 24 ranking spots worse, or what have you.

I'm on record here as saying I hate this kind of thing (and no, I didn't go to NYU) but I have to point out that most in the academy would say that NYU is a superior school to Columbia; certainly in terms of undergrad.

Posted by Freddie | May 20, 2007 8:09 PM

As a future elitist who went to an Ivy League university somewhat worse than Harvard but somewhat better than Brown

You accidentally included the word "future" in this statement.

Posted by aeroman | May 20, 2007 8:10 PM

I started out at an average local state school and transferred to a big prestigious out of state public school. So we're talking about the difference in rankings between 100+ and top-30. Honestly there really was not a big difference in terms of the material I came out knowing. There was not a big difference in starting salaries with my friends that stayed.

Sure, the latter school had better professors, more TAs, etc. But that didn't translate into a necessarily better education. The only quantifiable difference I could come up with in the end is that I ended up with about 3 times more debt than I would've had I stayed in-state. It's possible that name on the degree helped me land a job but really, I have no idea.

I don't think alumni income is a very good way to measure schools. First of all, obviously people go to school on a vastly different swath of majors. Second of all, after a few years on the job the significance of the name on that $50,000 piece of paper starts to become irrelevant.

Better to compare, on a program to program basis, the quality of teachers, the rigor of the curriculum, etc. But then, after all that, students may just find it's better to stay in-state and go to the cheapest school. Which is, honestly, the best option for about 90% of kids.

Posted by Joshua | May 20, 2007 8:12 PM

In addition to the problems Peter pointed out, the issue Fred identified makes alumni income a poor metric of educational performance. Employers use the university as a signaller not because they think it educated its students better, but because they trust its admissions decisions.

Posted by aeroman | May 20, 2007 8:20 PM

to me, US News rankings are worthless because the reputation of one's under grad institution isn't that important.

grad school is different. reputation matters.

But no medical school, law school, or graduate school will care what your UG institution is. test scores, gpa, letters or recommendation all take precedence.

Posted by thehova | May 21, 2007 12:46 AM

I can't believe this thread exists under the Atlantic's mast head.

http://www.theatlantic.com/doc/200311/confessore

Posted by crack | May 21, 2007 12:56 AM

I went to community college before I transfered to a, according to US News, top 20 university. Frankly the education I got was better at the community college, mainly because the class size was much smaller. It would be rare for a class to exceed 40 students by the middle of term. On the other hand classes at the University regularly exceeded 200 students.

Sure the University had top notch facilities and beyond brilliant professors, but that didn't matter much as most of the actual teaching was done by harried, sleep deprived grad students. In other words people who with some hard work, experience, and a bit of lucky might end up qualified to teach community college some day in the future.

The large class size made the classes easier too, too many students and tests to grade for creative or analytical thinking to be a critical factor. You had 2-4 different readers, so the criteria for grading had to be standardized. That resulted in study guides that basically gave away questions, you were guaranteed an excellent grade just for memorizing some outlines compiled out of class notes.

On the other hand, if I am going to devote years of my life to something, put in hard work, and pay for the privilege I better get more then just "an education", and on that count he University delivered.

And in that way US News is useful, I don't think most people use it as a guide to find the places of higher learning that give them the best education, but a guide that provides a ranking of the most prestigious schools. Just caring about the quality of education is a luxury of the affluent. The rest of us are interested in social mobility and see a university degree as a way to create more comfortable lives for ourselves and for our families.

Posted by Scyon | May 21, 2007 1:19 AM

OK, so I didn't go to an Ivy League university. Last I checked, USA Today ranked my alma mater somewhere in the 70s.

But we can probably beat your little preppie school in basketball.

And FWIW, in my experience thehova is right. Rankings are important for PhD and professional programs. Except maybe for a truly elite school like MIT or Cal Tech, your undergrad institution really doesn't matter much. A big state university and a small private college are completely different experiences that really can't be compared in any meaningful way, but you can get a quality education in either place.

Posted by LaFollette Progressive | May 21, 2007 1:39 AM

Any system of rankings will be open to endless debate. I suggest a 64 college single elimination tournament.

Posted by Njorl | May 21, 2007 2:08 AM

Currently, a group is trying to create value-added ratings for undergraduate colleges:

http://isteve.blogspot.com/2006/09/better-college-rankings.html

Posted by Steve Sailer | May 21, 2007 2:09 AM

For elite colleges where, say, half the students take the GRE, LSAT, or GMAT, it would be not unfeasible to compare average scores on those tests with SAT scores of entering freshmen (adjusted for percentage taking the postgrad education tests). That should give us an indication of value added.

My guess is that most colleges would come out fairly similar on value-added, with schools with high SAT scores for incoming freshmen also having high GRE/LSAT/GMAT scores for outgoing seniors. It would, however, be interesting to see if the small number of radically different programs, such as the two St. John's Great Books colleges, do better or worse than the mass of colleges.

Posted by Steve Sailer | May 21, 2007 2:16 AM

Any system of rankings will be open to endless debate. I suggest a 64 college single elimination tournament.

Who gets the #1 seeds? I'd say Yale, Princeton, MIT, and Brown. Harvard can be a #2 in the Brown bracket.

Posted by Al | May 21, 2007 2:22 AM

"It would, however, be interesting to see if the small number of radically different programs, such as the two St. John's Great Books colleges, do better or worse than the mass of colleges."

I remember getting the marketing material from St. Johns when I was in high school (a tag line something like "These are some of the professors returning to St. John's next year" followed by a dense list of great books authors such as Aristotle, Newton, etc.). I sometimes wish I had gone. One thing I remember from my research about the university is that they didn't release grades to students -- because they wanted to students to pursue learning as its own reward (they did keep some accounting of grades for when a grad school asked though). I wonder if this might make them hesitant to supply data to a list like this.

Posted by Dave | May 21, 2007 2:49 AM

Careful what you wish for.

I work in a portion of academia that has a very clear, reliably measured, & widely accepted metric. Medical schools evaluate themselves and their faculty based on NIH research grant $s received. (N of journal publications weighted by journal citation impact also matters, but less than it used to.) This isn't sour grapes, because I do well enough in that tournament. But the life world defined (revealed?) by that metric isn't what I was looking for when I was training.

Posted by Bill Gardner | May 21, 2007 7:52 AM

For elite colleges where, say, half the students take the GRE, LSAT, or GMAT, it would be not unfeasible to compare average scores on those tests with SAT scores of entering freshmen (adjusted for percentage taking the postgrad education tests).

I'm not sure that it's true that the more elite the college, the higher the percentage of students who go into postgraduate study.

Posted by Freddie | May 21, 2007 10:30 AM

I went to St. John's College and the basic reason that they don't supply any data is that they found themselves going up and down the lists without any change on their part.

If your being measure differently every year and nothing is changing other than the student body then I guess the measure isn't actually measuring what it says it is.

That and half the time they don't even have the information. I don't even think I sent them my SAT scores because I didn't have to.

Posted by Parmenides | May 21, 2007 2:47 PM

There are indeed easy ways for universities to game the system, in ways that benefit no one but the college itself. That said, there is a value in a third-party informing students--and the future employers or grad school educators of these students--about the relative quality of their undergraduate education. Ranking schools numerically is silly (is MIT four ranking spots better than CalTech, or five?) but placing schools in quality tiers would do a large amount of good without too much of the ankle-biting and ratings-grubbing that comes with the numerical ranking system.

Personally, I think the best system would stick with ordinal rankings, but have separate categories for each of a bunch of different factors. Say, a college might be 11th for livability (dorm accommodations, picturesque campus, proximity to a nice city, etc.), ninth for research (for the people who want to go to grad school there), 12th for undergraduate instruction, eighth for athletics and 14th for economic outcomes after graduation. Add more categories to evaluate as needed.

No system would be perfect, of course, but this would be a harder system to game than the USNWR system because it's not arbitrarily assigning values to each of those categories, like USNWR apparently does. And more generally, I think it would be more useful because it seems to genuinely reflect how people think, rather than assuming that everyone assigns exactly the same value to all parts of the college experience.

Posted by Cyrus | May 21, 2007 5:15 PM

Parmenides,

How did St. John's make admission decisions without SAT scores? I would imagine that a lot of students would wash out with that sort of curriculum if they weren't selected for aptitude in some way. Were there a lot of first year dropouts?

BTW, a question for you young ones: do your schools still use the term "freshmen"? When I went to school it was being replaced with "first year students" because of feminist concerns that the "men" in "freshmen" might be exclusionary or misogynist or something.

Posted by Dave | May 21, 2007 5:49 PM

How many people here think MY would be a famous pundit if he were equally talented, intelligent, and perceptive but hadn't gone to Harvard?

Or Yale, either.

Posted by SFG | May 22, 2007 5:45 AM

You have to write a long essay to get in. And yes there is a pretty high freshman washout rate. There is also the enabling process at the end of sophmore year where they kick out even more people.

Posted by Parmenides | May 23, 2007 9:23 AM