With predictions for the the 85th Annual Academy Awards flowing in, let’s revisit which film pundits fared best and worst with their Oscar predictions last year.  Here are our rankings for the 21 pundits we tracked, along with several of their predictions for this year (based on the latest available information, which we will update throughout the week).

Last Year’s Oscars: Rankings

Yield = how much money you would have made, on average, had you placed $1 bets on each of the pundit’s predictions (based on Vegas odds)

HR = Hit Rate = # correct predictions / # total predictions

Last year was a good one for the favorites (as deemed by oddsmakers), which outperformed most of the pundits we tracked on both a Yield and Hit Rate basis. For the pundits that scored well on our Yield metric, underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects accounted for most of the outperformance.

We will update the picks throughout the week and then provide updated rankings following Sunday’s results. If there are any other Oscar pundits that you wish to see tracked, just let us know.

Following the conclusion of the 84th Annual Academy Awards last night, it seemed only natural to turn our attention to the pundits who make their living predicting which Hollywood stars will come home with the world’s most infamous little gold statuettes. We hope the data below will prove informative, especially for those who spend countless hours trying to win their Oscar Pool with all kinds of techniques that range from picking the winners of the other awards shows to avoiding nominees that have already won. We would submit that the best technique may simply be to follow the pundit who has demonstrated time and again the ability to pick the most winners.

2012 is our first step in cataloguing those records. We chose 21 well-known film pundits, each of whom made predictions for all the Oscar categories.

Here are the rankings by hit rate (# of correct calls divided by # of total calls):

Hit Rate

These results are decidedly mediocre. Simply picking the favorites as deemed by Las Vegas oddsmakers (a proxy for crowd consensus) would have yielded a higher hit rate than 16 of our 21 pundits.

As we have discussed in previous posts, we believe the hit rate approach is flawed. All calls are not equal and should not be scored as such. Instead, pundits should receive more credit for predictions that are out-of-consensus. For instance, Christopher Plummer was a virtual lock for Best Supporting Actor, and so that pick should result in less credit than, say, choosing “Hugo” for Cinematography (over the favored “The Tree of Life”). Our “$1 Bet Yield” metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections. Here is how the Oscar pundits stacked up using this approach:

$1 Bet Yield

Here, the pundits fare a bit better, thanks largely to underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects. Still, the performance is uninspiring, with less than 40% of the group outperforming Vegas. 

Next year, we plan to track the awards predictions earlier in the process. The other award shows (Screen Actors Guild, Golden Globes, etc) tend to make the market more efficient, which in turn makes it more difficult to generate high Bet Yields come Oscar night, particularly in the major categories.

Update (2/27/12): Added one more pundit, Melena Ryzik of the Carpetbagger blog (New York Times).