Gladwellizing the college rankings

The US News college rankings get the Gladwell treatment in the current New Yorker (subscription required). The ease of parodying Gladwell is a basically good index of the ease of dismissing him, but this time I think he actually sheds light on a phenomenon, rather than just using the phenomenon as an occasion for doing his shtick. The upshot of the piece can be expressed very succinctly: the US News rankings are bunk.

And not just these rankings, but any ranking system that tries to be both homogeneous (ranking any and all members of the genus "college or university" regardless of species-level differences) and comprehensive (evaluating each one according to its total educational package, rather than on one aspect of it). Just as Car and Driver's comprehensive metric breaks down when you try to judge an $80,000 Lotus sports car and a $25,000 Hyundai minivan according to the same formula, so does US News produce such incongruous juxtapositions as Penn State and Yeshiva being right next to each other in the ranking of national universities, as if you could go to either one and have approximately the same educational experience. They're both good schools, after all.

Gladwell rightly observes that

There's no direct way to measure the quality of an institution--how well a college manages to inform, inspire, and challenge its students. So the US News algorithm relies instead on proxies for quality--and the proxies for educational quality turn out to be flimsy at best.

More about those proxies in a minute. You can't meaningfully compare Penn State and Yeshiva because they are only superficially trying to do the same thing. If Yeshiva started graduating students with the same values as the typical Nittany Lion, it will have utterly failed its stakeholders and mission. Likewise, a college chartered to educate "the sons of coalminers" is not set up to rake in $50,000 per head in tuition or build the massive endowment it takes to run an elite liberal arts college. To the extent that my college remains true to this founding mission, it will remain in its modest position in the status system that US News perpetuates: a system in which colleges are evaluated on the extent to which they accomplish Yale's mission. But King's might accomplish something else, providing a sort of social utility that Yale is not situated to provide.

We regularly evaluate other things relative to their aims; why not colleges? As Roger Ebert explains in his review of "Booty Call," a film he awarded three stars (out of four),

To evaluate this movie, I find myself falling back on my time-tested generic approach. First, I determine what the movie is trying to do, and what it promises its audiences they will see. Then, I evaluate how successful it is, and whether audiences will indeed see the movie they've been promised and enjoy it.

Ebert gave "The Fighter" two and a half stars. Is "The Fighter" (currently up for several Academy Awards) truly inferior to "Booty Call" (nominated for no Oscars in 1998)? Is it a worse piece of filmmaking? We might want to say no, but we can't discount that "The Fighter" is trying to be "Rocky" or "Raging Bull," reaching for immortality. "Booty Call" has, shall we say, no such aspirations. Ebert praised it for being everything its producers and its audience hoped it would be.

Given that colleges are presumably trying to cultivate students' faculties of judgment, it is not completely appropriate to say that they succeed when they give their "audience" exactly what they want. I hope, in fact, that colleges might be saved from what many prospective students want. But it's clear that colleges have differing educational missions and should be judged accordingly. At least, to the extent that we can judge their accomplishments at all.

While wines can be evaluated according to one or two variables and movies judged by what the viewer can plainly see, "educational quality" can at best be determined only impressionistically. So we have to look at proxies for quality. The single biggest factor in the US News formula is "undergraduate academic reputation." Which is established largely by ... the US News rankings themselves. To anyone inclined to be suspicious of systems of privilege, this feedback loop, engineered to reinforce status, is infuriating. (I'll say more about this in a future post.)

College presidents may lament the overemphasis on the US News rankings, but they also dutifully fill out the US News survey and await the results with the anxiety that accompanies any review of one's job performance. Could they--would they want to--break free of this codependency?

To short-circuit the rankings, college presidents would have to cooperate. But they are not wired to do so; they see other colleges as competitors, not collaborators in an educational mission. Princeton gets worried when Harvard makes an innovation like dispensing with loans. East Jahunga College worries that West Jahunga State will drop their foreign-language requirement. This massive prisoner's dilemma/Mexican standoff ensures that the system remains untouched, thereby boosting demand for anti-anxiety medication for teenagers, their parents, and college professors whose sense of self-worth is tied to the status of the place where they teach.

Fear, loathing, and trembling

Søren Kierkegaard wouldn't think much of an academic who was granted a promotion. Here's what he says about assistant professors:

How ludicrous an assistant professor is! We all laugh when a Mad Meyer tugs at a huge boulder which he believes is money - but the assistant professor goes around proudly, proud of his knowledge, and no one laughs. And yet that is just as ludicrous - to be proud of the knowledge by which a man dupes himself eternally.

Yes, you assistant professor, of all the loathsome inhumans the most loathsome, you may very well manage to say the same thing as the religious person has said, perhaps in even more beautiful language, you may also manage to reap worldly advantages with your shrewdness, yes, even honor and esteem such as the authentically religious person never one in this life - but you are duped eternally.

He's not much higher on associate professors:

Secured in life, they live in their thoughts; they have a permanent position and secure prospects in a well-organized state; they have centuries or indeed millennia between themselves and the earthquakes of existence... Their task in life is to judge the great men and to judge them according to the outcome. Such conduct toward the great betrays a curious mixture of arrogance and wretchedness - arrogance because they feel called to pass judgment, wretchedness because they do not feel their lives are even remotely related to those of the great.

Kierkegaard is usually both witty and withering in his criticism, but these strike me as pretty standard attacks on academics' vanity and isolation from a mythical land known as "the real world." If you've made it to associate professor, then you've faced this and worse. In fact, given the propensity for self-loathing among academics, most of us have said far worse things about own profession.

Only 16% of success is showing up

As a teacher of a humanities discipline, I’m not inclined to analyze things statistically. Wild claims, thinly supported? Now that’s more my style. But at the end of last semester, I realized that I had a convenient cache of data for helping figure out whether doing the reading really does contribute strongly to getting better grades in my classes – something I often tell my students, though I doubt many give these words much heed.

In both of my courses (each of which had two sections), I administered ten in-class quizzes. Most were unannounced. The quizzes generally tested basic knowledge of the reading assignments due on the day of the quiz. My ideal quiz is one that is extremely easy for a student who did the reading and extremely difficult for a student who didn’t. If the quizzes are at least reasonably well-designed, then I should be able to tell who read and who didn’t just by looking at the students’ grades on the quizzes.

The quizzes also serve as a way to spot-check attendance. Even though my classes are not huge (20-25 per section), I think that taking attendance every day wastes time and smacks a bit too much of high school. (It also requires better record-keeping than I’m really capable of.)

With the quiz data, then, I can tell how closely correlated attendance and reading are with overall performance in class. For each student, I calculated their grade on all written assignments other than quizzes, as I wanted to make quiz performance a truly independent variable. I also excluded class participation, which is partly determined by attendance. Correlating this grade with attendance was simple, as I could just tally up the number of quizzes a student took and compare it with their grade on all other assignments. Here’s what the graph of that looks like:

Stop worrying and learn to love worksheets?

To determine if your teaching is effective, you can craft assessments that truly show whether students are learning what you're teaching. You can also ask for instant feedback on the class session, a la the Frederick Mosteller one-minute paper. Or you can take the lazy professor's way out (this is preferred, obviously): you go by your intuitive sense of yourself and your students.

At the end of Monday's session in my MW class, Belief and Unbelief, things didn't feel quite right. As I packed up my multicolored markers and wished the last student to leave a good afternoon, I thought that I had talked too much, too abstractly, and not enough about the really important matters I'd hoped the students would get out of the class session. Even though by my tally, 17 of 24 students spoke up at some point during the class, it just didn't feel like I got enough of them to engage the gears of their minds with the axle of the central concepts I wanted them to master. I regretted distributing a NYT article about the search for "the God gene," which I did in hopes of highlighting the stakes involved in saying that religion is (or isn't) instinctual. That issue seemed a distraction from more important ones, such as making sure they understood what the main text for the day (Hume's Natural History of Religion) was actually saying.

Less verbigeration, more talking to teenagers

There is no shortage of arguments for preserving the liberal arts. A sampling of recent and prominent ones includes Gregory Petsko’s open letter to the President of SUNY-Albany, where several humanities departments will soon be folding; Eva von Dassow’s speech to the U. of Minnesota regents; Martha Nussbaum’s argument for privileging human development over economic development in educational efforts; and Louis Menand’s account of how humanities professors have turned their disciplines into professions, rather than into vehicles for developing transformative ideas. At National Review Online, Victor Davis Hanson proves that it’s not just politically liberal professors who think the liberal arts need to survive, defending the humanities’ central position in preserving cultural heritage.

All of these arguments are as excellent. They are also almost completely beside the point.

It’s true that to save departments that do not graduate future millionaires or produce patentable research, defenders of the humanities need to take their case to university administrators, boards of regents, state legislators, and the general public. At their best, the above examples attempt to convince these groups. At their worst, they are mere catechesis, attempting to convince people like me who already believe.

Are colleges like wines?

In America, each was once the near-exclusive preserve of wealthy, waspy elites but has more recently been democratized, marketed and made available to the middle and working classes. Though access to both of these complex goods is now universal, it is widely supposed that in both cases price tracks quality, such that the cost of a bottle of Dom Perignon or four years’ tuition at Brown is worth it: the product is better than one from a cheaper brand.

The problem is that when it comes to wine, price has little to do with quality, once quality is evaluated in truly blind taste tests. As Robin Goldstein shows in the book, The Wine Trials 2011 (which Goldstein co-edited with Alexis Herschkowitsch and Tyce Walters), Dom Perignon lost out in blind taste trials – including trials involving food and wine professionals – to the $12-a-bottle Domaine Ste. Michelle Brut. To Goldstein, what you’re really buying when you pay $150 for a bottle of Dom is the experience of owning and consuming something expensive. Buying Dom puts you in elite company: not just wealthy, but tastefully-yet-conspicuously wealthy.

Like wines, colleges are rated obsessively and vary widely in cost, and their quality is hard to pin down precisely, as many subjective elements are in play. Goldstein and his colleagues have developed a way to evaluate wines solely on what should be their most important aspect: how they taste. I wonder if a similar approach could be developed for colleges. If so, would it also bring to light some cheaper, lesser-regarded colleges that deliver high quality?

The state of the (liberal) arts

Will the liberal arts always be central to university education?  Are they now?  Certainly, they are becoming less central to SUNY-Albany, where several language and arts departments are being pruned (actually, made to wither and die, by executive decision) to save money.  It's not unreasonable to think that their roles will diminish at similar institutions, too.  The more apocalyptic among us might think that we are living in the last days, that this generation (of humanities professors) will not pass before it sees the last of their departments closed down.  Maybe that's going too far, but it is hard to look at the current state of the American university and think that the liberal arts have a bright future.  In fact, I take for granted that they don't.

Am I too pessimistic?  Perhaps the liberal arts are not dying, but merely changing.  Prophets of decline are easy to dismiss, in large part because they have so long been with us.  Perhaps the barbarians have simply always been at the gates, and they always end up being converted and setting up peacefully outside the castle.  The liberal arts have lasted this long, so why not assume they'll continue to last, if under a different, higher-tech form?