Harrison Hoffman at CNET thinks that the sudden elevation of The Dark Knight to the top film at IMDB illustrates a hole in the “wisdom of crowds” model. I think this is a bit off. For one, if we’re taking “the wisdom of crowds” to mean the bundle of concepts articulated in James Surowiecki’s book, then the caveats Hoffman is pointing out about the importance of structuring aggregation to prevent herding effects are already incorporated into the idea. And I’d always taken the point of peer-production of information to be that we accept the tradeoff in near-term accuracy for a process that yields better results over the long term: Don’t release right, release now. Presumably TDK will eventually slide to a more defensible spot on the list.
The better critique, I’d think, would point to the IMDB rankings without TDK. I realize, of course, that people can haggle endlessly over this or that omission, and at some point you’ve got to shrug and say de gustibus non disputandum—but there are some points where you really can say a list is just plainly indefensible. I’m not going to hold this up as a perfect alternative, but I think it’s instructive to compare the Village Voice‘s 100 Best Movies of the 20th Century list. Citizen Kane is at the top, which is where you tend to find it pretty much every time critics are asked to compile one of these lists. The number two movie on the IMDB list, The Shawshank Redemption, is (justly) nowhere to be found. And there’s not the heavy bias toward recent films (even taking into account that it’s explicitly a 20th century list) you find on the IMDB list.
Of course, critics lists have their own biases. Movies like Birth of a Nation and Steamboat Bill, Jr. are really up there because of their historical significance: In a vacuum, taken out of the context of the evolution of cinema, they’re plainly inferior to works ranked far below them. But critics can at least make their rankings relative to a huge body of films. The notable thing about the IMDB list is that, with a few exceptions, it’s heavily loaded with movies that the average person in their 20s or 30s is bound to have seen at some point. And this, it seems to me, runs somewhat counter to the purpose of a list like this. The idea of a “250 best films” ranking is, in large part, to give people some sense of what they might want to watch next time they’re filling up their Netflix queues. But what it’s ultimately picking up is that a lot more people have watched Pulp Fiction or The Matrix than have seen Rashomon or Berlin Alexanderplatz. And indeed, the former two are probably a better choice if you’re looking to have a few friends over for beer, popcorn, and a movie. But the IMDB ranking isn’t actually giving people much useful information: They’re up as high as they are precisely because everyone already knows about them.
So what can we say more generally? Again, most of this ground has been covered by James Surowiecki, but we can note that, above and beyond the herding problem, if you’re going to rely on “the wisdom of crowds” to rank a series of items, you need to account for different levels of exposure to the ranked items. An algorithm like IMDB’s might work somewhat better for music, since the average person has listened to many, many more songs than they’ve seen movies (which require a few hours of your undivided attention), but in both cases, you’d probably get better results by taking a page from John Stuart Mill, who recommended that pleasures be ranked by “competent judges” who are familiar with both. In other words, after gathering a bit of information about (say) genres or periods of movies a user is familiar with, IMDB could present a series of semi-randomized pairwise comparisons, along the lines of Facebook’s “Compare People” application. If and only if the user has seen both films, she simply chooses the one that she believes should be ranked higher.
Update: Commenter Christopher objects that this is just a misplaced demand for any “best movies” list to favor the “top 100 movies of the cultural elite”. Since I expect he won’t be alone in that reaction, I’ll copy my reply up here.
Certainly, as I point out above, part of the problem is that there are lots of different dimensions along which a movie can be “best”: Again, the best movie to sit and watch over beers with your friends, or to see on a first date, might not be the “best” in terms of elegant narrative and character development or innovative cinematography or whatever.
But first, even bracketing that, the exposure problem remains. Let’s say you just want the best action movie or the best kung-fu movie. A handful of people will have seen lots and lots and lots and have a decent basis for comparison; the majority will only be able to vote for the ones that are already well known. If this yields results that aren’t wildly off, it’s going to be primarily because what gets to be well known over time already incorporates a certain amount of expert evaluation.
Second, I want to push back at least somewhat against the idea that the “cultural elite” preference is just some kind of niche demographic taste, like “what’s popular in Omaha”. People who are really really into movies display a degree of convergence about which are the great films, and while this may in part be attributable to groupthink, in part it’s because they’re picking up on powerful visual or narrative techniques that anyone could appreciate with a well cultivated eye. Being part of the “cultural elite” just is developing the kind of eye that responds to more complex and original (or, if you will, “better”) visual storytelling. If what’s constitutive of your niche demographic is that it’s composed of people who are better attuned to good filmmaking, then yeah, their preferences probably should be more salient. Not because they’re inherently wonderful people, but because they’re further along the curve that you, too, average moviegoer, would traverse if you watched many movies attentively.