Gladwellizing the college rankings

The US News college rankings get the Gladwell treatment in the current New Yorker (subscription required). The ease of parodying Gladwell is a basically good index of the ease of dismissing him, but this time I think he actually sheds light on a phenomenon, rather than just using the phenomenon as an occasion for doing his shtick. The upshot of the piece can be expressed very succinctly: the US News rankings are bunk.

And not just these rankings, but any ranking system that tries to be both homogeneous (ranking any and all members of the genus "college or university" regardless of species-level differences) and comprehensive (evaluating each one according to its total educational package, rather than on one aspect of it). Just as Car and Driver's comprehensive metric breaks down when you try to judge an $80,000 Lotus sports car and a $25,000 Hyundai minivan according to the same formula, so does US News produce such incongruous juxtapositions as Penn State and Yeshiva being right next to each other in the ranking of national universities, as if you could go to either one and have approximately the same educational experience. They're both good schools, after all.

Gladwell rightly observes that

There's no direct way to measure the quality of an institution--how well a college manages to inform, inspire, and challenge its students. So the US News algorithm relies instead on proxies for quality--and the proxies for educational quality turn out to be flimsy at best.

More about those proxies in a minute. You can't meaningfully compare Penn State and Yeshiva because they are only superficially trying to do the same thing. If Yeshiva started graduating students with the same values as the typical Nittany Lion, it will have utterly failed its stakeholders and mission. Likewise, a college chartered to educate "the sons of coalminers" is not set up to rake in $50,000 per head in tuition or build the massive endowment it takes to run an elite liberal arts college. To the extent that my college remains true to this founding mission, it will remain in its modest position in the status system that US News perpetuates: a system in which colleges are evaluated on the extent to which they accomplish Yale's mission. But King's might accomplish something else, providing a sort of social utility that Yale is not situated to provide.

We regularly evaluate other things relative to their aims; why not colleges? As Roger Ebert explains in his review of "Booty Call," a film he awarded three stars (out of four),

To evaluate this movie, I find myself falling back on my time-tested generic approach. First, I determine what the movie is trying to do, and what it promises its audiences they will see. Then, I evaluate how successful it is, and whether audiences will indeed see the movie they've been promised and enjoy it.

Ebert gave "The Fighter" two and a half stars. Is "The Fighter" (currently up for several Academy Awards) truly inferior to "Booty Call" (nominated for no Oscars in 1998)? Is it a worse piece of filmmaking? We might want to say no, but we can't discount that "The Fighter" is trying to be "Rocky" or "Raging Bull," reaching for immortality. "Booty Call" has, shall we say, no such aspirations. Ebert praised it for being everything its producers and its audience hoped it would be.

Given that colleges are presumably trying to cultivate students' faculties of judgment, it is not completely appropriate to say that they succeed when they give their "audience" exactly what they want. I hope, in fact, that colleges might be saved from what many prospective students want. But it's clear that colleges have differing educational missions and should be judged accordingly. At least, to the extent that we can judge their accomplishments at all.

While wines can be evaluated according to one or two variables and movies judged by what the viewer can plainly see, "educational quality" can at best be determined only impressionistically. So we have to look at proxies for quality. The single biggest factor in the US News formula is "undergraduate academic reputation." Which is established largely by ... the US News rankings themselves. To anyone inclined to be suspicious of systems of privilege, this feedback loop, engineered to reinforce status, is infuriating. (I'll say more about this in a future post.)

College presidents may lament the overemphasis on the US News rankings, but they also dutifully fill out the US News survey and await the results with the anxiety that accompanies any review of one's job performance. Could they--would they want to--break free of this codependency?

To short-circuit the rankings, college presidents would have to cooperate. But they are not wired to do so; they see other colleges as competitors, not collaborators in an educational mission. Princeton gets worried when Harvard makes an innovation like dispensing with loans. East Jahunga College worries that West Jahunga State will drop their foreign-language requirement. This massive prisoner's dilemma/Mexican standoff ensures that the system remains untouched, thereby boosting demand for anti-anxiety medication for teenagers, their parents, and college professors whose sense of self-worth is tied to the status of the place where they teach.