We tracked 22 pundits who made predictions for all 24 categories of the 85th Annual Academy Awards. Here are the results:
[The second column is the # of predictions that turned out correct. The third column is the Yield, which measures the average payout, based on Vegas odds, had you bet $1 on each of the predictions. This gives pundits more credit for out-of-consensus calls, and a Yield above $1.00 means the pundit did better than the consensus view]
Pundit Hit Rate Yield
Anthony Breznican, Entertainment Weekly 21/24 $1.61
Scott Feinberg, Hollywood Reporter 21/24 $1.48
Anne Thompson, Indiewire 20/24 $1.45
Melena Ryzik, New York Times 20/24 $1.31
Gary Susman, MovieFone 17/24 $1.22
Clayton Davis, The Awards Circuit 17/24 $1.12
Kris Tapley, HitFix 18/24 $1.08
Richard Lawson, The Atlantic Wire 18/24 $1.07
Tom Shone, The Guardian 18/24 $1.05
Vegas Favorites 19/24 $1.03
Joel D Amos, Movie Fanatic 17/24 $1.02
Rotten Tomatoes 18/24 $1.01
Glenn Whipp, Los Angeles Times 17/24 $0.96
Peter Knegt, Indiewire 17/24 $0.94
Guy Lodge, HitFix 16/24 $0.93
Richard Roeper, Chicago Sun-Times 16/24 $0.89
Joal Ryan, E! Online 15/24 $0.87
Pete Hammond, Deadline.com 15/24 $0.86
Kevin Polowy, NextMovie 16/24 $0.84
Steve Pond, TheWrap 15/24 $0.83
Kyle Buchanan, Vulture 16/24 $0.82
Wesley Morris, Grantland 11/24 $0.69
The Best Pundit award goes to Anthony Breznican of Entertainment Weekly, who got 21/24 picks correct, including non-consensus ones such as “Lincoln” for Production Design and “Skyfall” for Sound Editing (where it earned a tie). Scott Feinberg of the Hollywood Reporter and Anne Thompson of Indiewire also scored high marks, while Wesley Morris of Grantland had a rough showing. Simply going with the favorites — as deemed by the oddsmakers — would have yielded a slightly positive return.
Finally, here are the cumulative rankings of the pundits we track that made predictions for all categories both last year and this year, with Melena Ryzik of the Carpetbagger blog (New York Times) earning the top spot.
Pundit Last Year This Year Cumulative
Melena Ryzik, New York Times $1.36 $1.31 $1.34
Anne Thompson, Indiewire $1.07 $1.45 $1.26
Clayton Davis, The Awards Circuit $1.35 $1.12 $1.24
Kris Tapley, HitFix $1.39 $1.08 $1.24
Scott Feinberg, Hollywood Reporter $0.96 $1.48 $1.22
Peter Knegt, Indiewire $1.37 $0.94 $1.16
Vegas Favorites $1.10 $1.03 $1.07
Glenn Whipp, Los Angeles Times $1.16 $0.96 $1.06
Pete Hammond, Deadline.com $1.22 $0.86 $1.04
Steve Pond, TheWrap $1.14 $0.83 $0.99
Guy Lodge, HitFix $1.01 $0.93 $0.97
Rotten Tomatoes $0.83 $1.01 $0.92
Kyle Buchanan, Vulture $1.01 $0.82 $0.92
Kevin Polowy, NextMovie $0.95 $0.84 $0.90
Richard Roeper, Chicago Sun-Times $0.73 $0.89 $0.81
Update: We originally had Kris Tapley scored at 17/24, which was incorrect. This has been fixed.
With predictions for the the 85th Annual Academy Awards flowing in, let’s revisit which film pundits fared best and worst with their Oscar predictions last year. Here are our rankings for the 21 pundits we tracked, along with several of their predictions for this year (based on the latest available information, which we will update throughout the week).
Last Year’s Oscars: Rankings
Yield = how much money you would have made, on average, had you placed $1 bets on each of the pundit’s predictions (based on Vegas odds)
HR = Hit Rate = # correct predictions / # total predictions
Yield HR Picture Actor Actress S Actor S Actress Director
Kris Tapley $1.39 79% Argo DDL Riva DeNiro Hathaway Lee
Peter Knegt $1.37 79% Argo DDL Riva DeNiro Hathaway Spielberg
Clayton Davis $1.35 83% Argo DDL Riva Waltz Hathaway Russell
Melena Ryzik $1.36 83% Argo DDL Lawrence DeNiro Hathaway Lee
Pete Hammond $1.22 75% Argo DDL Riva DeNiro Hathaway Lee
Glenn Whipp $1.16 79% Steve Pond $1.14 75% Argo DDL Riva DeNiro Hathaway Lee
Sasha Stone $1.11 75% Vegas Favorites $1.10 77% Argo DDL Lawrence Jones Hathaway Spielberg
Dave Karger $1.09 75% Argo DDL Lawrence DeNiro Hathaway Lee
Anne Thompson $1.07 71% Argo DDL Lawrence Jones Hathaway Lee
Guy Lodge $1.01 67% Argo DDL Riva Waltz Hathaway Lee
Kyle Buchanan $1.01 67% Argo DDL Lawrence Jones Hathaway Spielberg
Lou Lumenick $0.98 65% Jim Slotek $0.98 58% Scott Feinberg $0.96 67% Argo DDL Lawrence Waltz Hathaway Lee
Kevin Polowy $0.95 67% Argo DDL Lawrence Jones Hathaway Spielberg
Rotten Tomatoes $0.83 58% Argo DDL Lawrence Jones Hathaway Spielberg
Mark Harris $0.76 58% Richard Roeper $0.73 58% Argo DDL Lawrence DeNiro Hathaway Spielberg
Erik Childress $0.63 54% S.T. VanAirsdale $0.59 43%
Last year was a good one for the favorites (as deemed by oddsmakers), which outperformed most of the pundits we tracked on both a Yield and Hit Rate basis. For the pundits that scored well on our Yield metric, underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects accounted for most of the outperformance.
We will update the picks throughout the week and then provide updated rankings following Sunday’s results. If there are any other Oscar pundits that you wish to see tracked, just let us know.
In our last post, we highlighted our preference for “second acts”, which we defined as repeated success in different environments. We focused on repeated success in that discussion. Here, we will delve into the second component of the definition: different environments.
As with broken clocks, which are right twice each day, many pundits make the same prediction over and over, knowing that they will be intermittently correct. The stock market offers an ideal breeding ground for this type of pundit, given its long cycles and binary outcomes (the market is either up or down). Those who are always optimistic on the stock market (known as “perma-bulls”) are right during bull markets, while those who are always downbeat (“perma-bears”) are right during bear markets. Both groups are anointed as gurus when the environment happens to be in their favor, but they are dead wrong during the other cycle. The behavioral elements discussed in our One-Hit Wonders post are again at play: recent, vivid, and unusual information are front-loaded in our brains. But another bias comes into play here: the “fundamental attribution error.”
The fundamental attribution error refers to the idea that when explaining successes and failures, we tend to overweight the role of the individual and underweight the roles of chance and context. For instance, in the early 2000s, much ink was spilled in lauding Terry Semel and Meg Whitman (then CEOs of Yahoo and eBay, respectively) as superstar executives. The alternative explanation — that these CEOs were running companies that happened to be at the right place at the right time — never received much thought. That’s not surprising, given that cause-and-effect, character-driven narratives are inherently much more appealing than chalking things up to good fortune.
Similarly, rather than considering that some pundits offer a static viewpoint which happens to coincide with an existing cycle, we elevate them to oracle status. We hope PunditTracker will help distinguish those pundits with the mental flexibility to provide insight in different environments from the broken clock crowd.
At PunditTracker, we place extra weight on second acts: people who demonstrate repeated success in different environments. Examples include former Apple CEO Steve Jobs and NFL coach Bill Parcells.
Just as half-truths are more dangerous than outright lies, the most dangerous pundit is the one who parlays a single correct call into guru status. While the media plays a central role in this game, our brains are culpable as well. Behavioral studies reveal that certain types of information have an outsized grip on our memory, including that which is recent, vivid, and unusual. This explains why we: (1) buy earthquake insurance after a (recent) earthquake, (2) believe that more people die from homicide (vivid) than from stomach cancer, (3) complain that we always get stuck in the slowest-moving line (unusual) at the grocery store.
The pundit playbook fully exploits this dynamic. To understand how, let’s place ourselves in the shoes of a pundit. What is the best way to make a call that meets all three criteria: recent, vivid, and unusual? Well, recent is easy—just make a lot of calls. That way, there will always be a fresh one out there. And vivid is just another word for bold. So if we make bold calls frequently, we are two-thirds of the way there. But how about unusual?
The beauty of the pundit playbook is that unusual takes care of itself. Bold calls are typically incorrect, so the correct ones are by definition unusual. Said differently, bold calls that turn out wrong are less likely to be remembered because they fail to meet the unusual threshold. Pundits are therefore entirely incentivized to churn out brash predictions, knowing that only the correct ones will stick in our mind. And because we tend to confuse ease of recall with frequency, we develop a warped sense of the pundit’s batting average.
This phenomenon is found in all walks of life, including sports. NBA guard Chauncey Billups, for instance, has been dubbed “Mr. Big Shot,” presumably because he has hit many clutch shots in his career. A closer look at the numbers, however, suggests that Billups’ nickname might be undeserved. Data from 82games.com reveals that Billups’ game-winning shot percentage between 2003 and 2008 was a paltry 16% (6 for 37), well below his 42% overall career shooting average. Our hunch is that a few of those game-winners were in high-profile, nationally televised games (vivid), thus sticking in the public’s mind and inflating Billups’ reputation as a clutch shooter.
The real danger comes when actions are taken based on a false premise. In this case, Billups’ inflated reputation is likely to garner him most of his team’s game-winning shot attempts, even though other players would be better options. Teammate Carmelo Anthony, for instance, had a sterling 48% game-winning shot percentage (13 for 27) over the same timeframe.
As anyone in marketing knows, once established, associations are very sticky. This explains how pundits are able to cash in for many years on the “one big call” they got right, despite sporting a terrible track record both before and afterwards. By playing the role of public scorekeeper, we hope that surfacing the data for everyone to see will help expose the one-hit wonders.
We are thrilled to announce the launch of the PunditTracker.com website. The blog will continue in its current form, serving to contextualize, analyze, and discuss the content on the main site.
Let us know what you think!
Following the conclusion of the 84th Annual Academy Awards last night, it seemed only natural to turn our attention to the pundits who make their living predicting which Hollywood stars will come home with the world’s most infamous little gold statuettes. We hope the data below will prove informative, especially for those who spend countless hours trying to win their Oscar Pool with all kinds of techniques that range from picking the winners of the other awards shows to avoiding nominees that have already won. We would submit that the best technique may simply be to follow the pundit who has demonstrated time and again the ability to pick the most winners.
2012 is our first step in cataloguing those records. We chose 21 well-known film pundits, each of whom made predictions for all the Oscar categories.
Here are the rankings by hit rate (# of correct calls divided by # of total calls):
|Clayton Davis, The Awards Circuit||83.3%|
|Melena Ryzik, New York Times||83.3%|
|Kris Tapley, HitFix||79.2%|
|Peter Knegt, Indiewire||79.2%|
|Glenn Whipp, The Envelope||79.2%|
|Pete Hammond, Deadline.com||75.0%|
|Steve Pond, TheWrap||75.0%|
|Sasha Stone, Awards Daily||75.0%|
|Dave Karger, Entertainment Weekly||75.0%|
|Anne Thompson, Indiewire||70.8%|
|Guy Lodge, HitFix||66.7%|
|Kyle Buchanan, Vulture||66.7%|
|Scott Feinberg, Hollywood Reporter||66.7%|
|Kevin Polowy, NextMovie||66.7%|
|Lou Lumenick, New York Post||65.2%|
|Jim Slotek, Toronto Sun||58.3%|
|Mark Harris, Grantland||58.3%|
|Richard Roeper, Chicago Sun-Times||58.3%|
|Erik Childress, Movies.com||54.2%|
These results are decidedly mediocre. Simply picking the favorites as deemed by Las Vegas oddsmakers (a proxy for crowd consensus) would have yielded a higher hit rate than 16 of our 21 pundits.
As we have discussed in previous posts, we believe the hit rate approach is flawed. All calls are not equal and should not be scored as such. Instead, pundits should receive more credit for predictions that are out-of-consensus. For instance, Christopher Plummer was a virtual lock for Best Supporting Actor, and so that pick should result in less credit than, say, choosing “Hugo” for Cinematography (over the favored “The Tree of Life”). Our “$1 Bet Yield” metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections. Here is how the Oscar pundits stacked up using this approach:
$1 Bet Yield
|Kris Tapley, HitFix||$1.39|
|Peter Knegt, Indiewire||$1.37|
|Clayton Davis, The Awards Circuit||$1.35|
|Melena Ryzik, New York Times||$1.35|
|Pete Hammond, Deadline.com||$1.22|
|Glenn Whipp, The Envelope||$1.16|
|Steve Pond, TheWrap||$1.14|
|Sasha Stone, Awards Daily||$1.11|
|Dave Karger, Entertainment Weekly||$1.09|
|Anne Thompson, Indiewire||$1.07|
|Guy Lodge, HitFix||$1.01|
|Kyle Buchanan, Vulture||$1.01|
|Lou Lumenick, New York Post||$0.98|
|Jim Slotek, Toronto Sun||$0.98|
|Scott Feinberg, Hollywood Reporter||$0.96|
|Kevin Polowy, NextMovie||$0.95|
|Mark Harris, Grantland||$0.76|
|Richard Roeper, Chicago Sun-Times||$0.73|
|Erik Childress, Movies.com||$0.63|
|S.T. VainAirsdale, MovieLine||$0.59|
Here, the pundits fare a bit better, thanks largely to underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects. Still, the performance is uninspiring, with less than 40% of the group outperforming Vegas.
Next year, we plan to track the awards predictions earlier in the process. The other award shows (Screen Actors Guild, Golden Globes, etc) tend to make the market more efficient, which in turn makes it more difficult to generate high Bet Yields come Oscar night, particularly in the major categories.
Update (2/27/12): Added one more pundit, Melena Ryzik of the Carpetbagger blog (New York Times).