Following the conclusion of the 84th Annual Academy Awards last night, it seemed only natural to turn our attention to the pundits who make their living predicting which Hollywood stars will come home with the world’s most infamous little gold statuettes. We hope the data below will prove informative, especially for those who spend countless hours trying to win their Oscar Pool with all kinds of techniques that range from picking the winners of the other awards shows to avoiding nominees that have already won. We would submit that the best technique may simply be to follow the pundit who has demonstrated time and again the ability to pick the most winners.
2012 is our first step in cataloguing those records. We chose 21 well-known film pundits, each of whom made predictions for all the Oscar categories.
Here are the rankings by hit rate (# of correct calls divided by # of total calls):
Hit Rate
| Clayton Davis, The Awards Circuit | 83.3% |
| Melena Ryzik, New York Times | 83.3% |
| Kris Tapley, HitFix | 79.2% |
| Peter Knegt, Indiewire | 79.2% |
| Glenn Whipp, The Envelope | 79.2% |
| Vegas favorites | 77.1% |
| Pete Hammond, Deadline.com | 75.0% |
| Steve Pond, TheWrap | 75.0% |
| Sasha Stone, Awards Daily | 75.0% |
| Dave Karger, Entertainment Weekly | 75.0% |
| Anne Thompson, Indiewire | 70.8% |
| Guy Lodge, HitFix | 66.7% |
| Kyle Buchanan, Vulture | 66.7% |
| Scott Feinberg, Hollywood Reporter | 66.7% |
| Kevin Polowy, NextMovie | 66.7% |
| Lou Lumenick, New York Post | 65.2% |
| Jim Slotek, Toronto Sun | 58.3% |
| Rotten Tomatoes | 58.3% |
| Mark Harris, Grantland | 58.3% |
| Richard Roeper, Chicago Sun-Times | 58.3% |
| Erik Childress, Movies.com | 54.2% |
These results are decidedly mediocre. Simply picking the favorites as deemed by Las Vegas oddsmakers (a proxy for crowd consensus) would have yielded a higher hit rate than 16 of our 21 pundits.
As we have discussed in previous posts, we believe the hit rate approach is flawed. All calls are not equal and should not be scored as such. Instead, pundits should receive more credit for predictions that are out-of-consensus. For instance, Christopher Plummer was a virtual lock for Best Supporting Actor, and so that pick should result in less credit than, say, choosing “Hugo” for Cinematography (over the favored “The Tree of Life”). Our “$1 Bet Yield” metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections. Here is how the Oscar pundits stacked up using this approach:
$1 Bet Yield
| Kris Tapley, HitFix | $1.39 |
| Peter Knegt, Indiewire | $1.37 |
| Clayton Davis, The Awards Circuit | $1.35 |
| Melena Ryzik, New York Times | $1.35 |
| Pete Hammond, Deadline.com | $1.22 |
| Glenn Whipp, The Envelope | $1.16 |
| Steve Pond, TheWrap | $1.14 |
| Sasha Stone, Awards Daily | $1.11 |
| Vegas favorites | $1.10 |
| Dave Karger, Entertainment Weekly | $1.09 |
| Anne Thompson, Indiewire | $1.07 |
| Guy Lodge, HitFix | $1.01 |
| Kyle Buchanan, Vulture | $1.01 |
| Lou Lumenick, New York Post | $0.98 |
| Jim Slotek, Toronto Sun | $0.98 |
| Scott Feinberg, Hollywood Reporter | $0.96 |
| Kevin Polowy, NextMovie | $0.95 |
| Rotten Tomatoes | $0.83 |
| Mark Harris, Grantland | $0.76 |
| Richard Roeper, Chicago Sun-Times | $0.73 |
| Erik Childress, Movies.com | $0.63 |
| S.T. VainAirsdale, MovieLine | $0.59 |
Here, the pundits fare a bit better, thanks largely to underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects. Still, the performance is uninspiring, with less than 40% of the group outperforming Vegas.
Next year, we plan to track the awards predictions earlier in the process. The other award shows (Screen Actors Guild, Golden Globes, etc) tend to make the market more efficient, which in turn makes it more difficult to generate high Bet Yields come Oscar night, particularly in the major categories.
Update (2/27/12): Added one more pundit, Melena Ryzik of the Carpetbagger blog (New York Times).
