Good news! We are putting the finishing touches on PunditTracker and expect to launch the site next month (prediction alert!). The blog posts will continue to be sparse in the coming weeks, but we expect the blog will serve as an integral part of the website post-launch.
We launched the blog before the website in order to gather user feedback and adjust accordingly. The enthusiastic response has been encouraging. We are grateful for all your input and suggestions, many of which have been incorporated into the website.
As we have referenced in previous posts, the success of PunditTracker will largely come down to user engagement. While we will continue to do our best to find and track pundit predictions, scalability will eventually become a challenge, so we will need your help. If you come across any new predictions, please send them our way and we will post them on the site.
We graded the 2012 Round 1 NFL mock drafts of Mel Kiper (ESPN), Todd McShay (ESPN), and Mike Mayock (NFL.com), and once again, McShay came out on top. This marks the fourth consecutive year in which McShay bested his two fellow pundits in mock draft accuracy.
As referenced in our previous post, we use a variability measure based on the difference between when each player was projected to be picked and when he was actually picked. A lower variability score is better (ranks in parentheses).
| Variability | 2009 | 2010 | 2011 | 2012 | Average |
| McShay | 8.7 (1) | 8.8 (1) | 6.8 (1) | 7.0 (1) | 7.8 (1) |
| Kiper | 8.8 (2) | 10.4 (3) | 7.7 (2) | 8.5 (3) | 8.9 (2) |
| Mayock | 9.6 (3) | 9.9 (2) | 9.3 (3) | 7.9 (2) | 9.1 (3) |
We calculated two additional metrics as well: (1) a “weighted variability” measure, which penalizes a pundit more for getting earlier picks wrong, and (2) a point score system, which diminishes the impact of extremely wrong predictions by capping the downside penalty. The year-to-year rankings vary materially across the three measures, but the overall story is the same: McShay ranks #1 on average, regardless of the metric used. We have also updated the major hits and misses for the group.
Here is a guest post from a friend of PunditTracker.
Former Pennsylvania Senator Rick Santorum suspended his presidential campaign Tuesday, bowing to the inevitability of Mitt Romney’s nomination. The decision came in the wake of three solid wins by the former Bain Capital CEO last week in Maryland, the District of Columbia and Wisconsin.
As the pace of the nominating horse race has lagged, the media has pivoted predictably to what the Washington Post’s Chris Cillizza described as “the great process that is the vice presidential sweepstakes.” “It is never too early,” quipped CNN’s Candy Crowley, “for the political world to play Veepstakes.”
On its face, the vice presidential selection storyline is a meritorious one. Jules Witcover argued in the acclaimed Crapshoot that the first most important decision a president makes comes well before taking office in the choice of a vice presidential nominee. And so it does.
The party standard bearer, in contemplating a running mate, is weighing two frequently competing imperatives: improving their prospects of electoral success and running the government if elected. In the last decade, Sarah Palin, John Edwards and Joe Lieberman exemplified the former ambition. Joe Biden and Dick Cheney represented the latter.
As Romney looks ahead to Tampa, a set of clear general election headwinds loom: weakness with the conservative base; regional challenges in the South; and a Curriculum Vitae ill suited to harnessing prevalent anti-Wall Street sentiment. His selection may seek to address those gaps. Alternatively, he may look to buttress a business narrative that has been muted by a bruising GOP nominating process with Senator Santorum.
Over the weekend, conservative commentator George F. Will weighed in on behalf of a selection with “intellectual firepower, born of immersion in policy complexities.” Will championed two current leaders: WI Congressman Paul Ryan or Louisiana Governor Bobby Jindal.
Ryan and Jindal, for their part, have been consistent media mentions on the so-called pre-convention “long short list” that includes FL Senator Marco Rubio, Ohio Senator Rob Portman, New Jersey Governor Chris Christie, Indiana Governor Mitch Daniels and VA Governor Tom McDonnell. The Post’s Cillizza asserted Sunday that if you pick Rubio, Jindal or Christie, “you have a 33 percent chance of telling all of your friends that you picked the vice presidential nominee months before Romney announced it.”
Amid the selection certitudes from Cillizza and others, however, one can’t help but recall Richard Nixon and the choice of then-Maryland Governor Spiro Agnew as his running mate in1968. A stunned press corps responded at the announcement of Agnew with quizzical shouts of “Spiro Who?” 1988 (Quayle), 1996 (Kemp), 2000 (Cheney) and 2008 (Palin) offer similar cause for speculative caution.
But the modern media landscape rewards bold prognostications, largely independent of outcome. The cycle-over-cycle difference is that PunditTracker will be chronicling this Veepstakes, reminding us who gets it right – and who gets it wrong.
Donald Trigg serves as the Chief Revenue Officer for CodeRyte, the leading Natural Language Processing (NLP) supplier in healthcare. He is a veteran of several presidential pursuits, including his work on the 2000 Bush for President campaign in Austin, Texas.
Message from PunditTracker: Send us any VP predictions you come across! Either click the button at the bottom of the page or e-mail us at admin@pundittracker.com.
Here are our final 2012 March Madness bracket rankings, updated using the Final Four and National Champion futures odds. Since we did not have futures odds for the Elite 8 or the Championship game, those two rounds are not included in our calculations.
Peter Tiernan of CBS narrowly edged out Jay Bilas and President Obama in the $1 Bet Yield rankings, while Bilas kept his lead in the Hit Rate rankings. While none of the pundits picked Louisville to make the Final Four, three of them (Rob Dauster, Jerry Palm, and Brad Evans) had the other three teams correct. 24 of the 26 pundits had Kentucky winning it all.
$1 Bet Yield
| Peter Tiernan, CBS | $1.23 |
| Jay Bilas, ESPN | $1.22 |
| Barack Obama | $1.22 |
| Rob Dauster, SI | $1.21 |
| Nick Lachey | $1.20 |
| Jeff Goodman, CBS | $1.20 |
| Jeff Borzello, CBS | $1.08 |
| Jerry Palm, CBS | $1.03 |
| Colin Cowherd, ESPN | $1.02 |
| Matt Norlander, CBS | $1.01 |
| Luke Winn, SI | $0.99 |
| Andy Staples, SI | $0.96 |
| Joe Lunardi, ESPN | $0.95 |
| Gary Parrish, CBS | $0.93 |
| Greg Doyel, CBS | $0.92 |
| Dan Wetzel, Yahoo | $0.91 |
| Pat Forde, Yahoo | $0.90 |
| Dennis Dodd, CBS | $0.87 |
| Andy Katz, ESPN | $0.87 |
| Brad Evans, Yahoo | $0.87 |
| Dick Vitale, ESPN | $0.86 |
| Stewart Mandel, SI | $0.86 |
| Andy Glockner, SI | $0.86 |
| Seth Davis, SI | $0.85 |
| LeBron James | $0.81 |
| Mike Huguenin, Yahoo | $0.80 |
Hit Rate
| Jay Bilas, ESPN | 73.6% |
| Barack Obama | 71.7% |
| Peter Tiernan, CBS | 69.8% |
| Nick Lachey | 69.8% |
| Rob Dauster, SI | 67.9% |
| Jeff Goodman, CBS | 67.9% |
| Colin Cowherd, ESPN | 64.2% |
| Jeff Borzello, CBS | 62.3% |
| Jerry Palm, CBS | 62.3% |
| Dan Wetzel, Yahoo | 62.3% |
| Luke Winn, SI | 60.4% |
| Andy Staples, SI | 60.4% |
| Joe Lunardi, ESPN | 60.4% |
| Gary Parrish, CBS | 60.4% |
| Matt Norlander, CBS | 58.5% |
| Dick Vitale, ESPN | 58.5% |
| Andy Katz, ESPN | 56.6% |
| Stewart Mandel, SI | 56.6% |
| Andy Glockner, SI | 56.6% |
| Greg Doyel, CBS | 54.7% |
| Pat Forde, Yahoo | 54.7% |
| Dennis Dodd, CBS | 54.7% |
| Brad Evans, Yahoo | 54.7% |
| Seth Davis, SI | 54.7% |
| LeBron James | 52.8% |
| Mike Huguenin, Yahoo | 52.8% |
As we stated in an earlier post, we have been tracking Jim Cramer’s Mad Money featured stock picks since January 2011 and will be providing an update on the returns each month. We assume a one-year holding period for each of his picks (unless Cramer reverses his opinion on the security, in which case we trade accordingly) and consider a “sell” rating to be a recommendation to short the stock. Some of our readers have argued that our one-year holding period assumption is too long, so in this post, we are also providing the returns on a three-month and six-month holding period basis.
Here is: (1) the average return for each holding period, (2) the corresponding annualized return to allow for comparability, (3) our $1 Yield metric, and (4) the Normalized Yield, which is simply Cramer’s $1 Yield divided by the S&P’s $1 Yield.
| One year (61 picks) | Return | Annualized | $1 Yield | Normalized |
| Cramer | -11.1% | -11.1% | $0.89 | $0.88 |
| S&P | 1.4% | 1.4% | $1.01 | |
| Six month (168) | ||||
| Cramer | -6.2% | -12.0% | $0.88 | $0.92 |
| S&P | -2.2% | -4.3% | $0.96 | |
| Three month (247) | ||||
| Cramer | 0.2% | 0.9% | $1.01 | $0.94 |
| S&P | 1.9% | 7.7% | $1.08 |
The normalized yield measures relative performance. If this figure is greater than $1.00, then Cramer has outperformed the S&P; less than $1.00 reflects underperformance. For the picks tracked to date, Cramer has underperformed regardless of the holding period assumption.
With the first two rounds of March Madness complete, let’s see how the pundits fared with their brackets. We scored pundits from ESPN, Yahoo, Sports Illustrated, and CBS Sports, throwing in President Obama, LeBron James, and Nick Lachey for good fun (all three made their brackets available on ESPN).
As a reminder, our “$1 Bet Yield” metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections. This approach fits well with March Madness, since it mimics bracket scoring systems that award extra points for predicting upsets. We used standard Vegas odds for Round 1 and Sweet 16 futures odds for Round 2 (courtesy of sportsbook 5Dimes).
$1 Bet Yield
| Jeff Goodman, CBS | $1.22 |
| Peter Tiernan, CBS | $1.19 |
| Jay Bilas, ESPN | $1.19 |
| Barack Obama | $1.19 |
| Nick Lachey | $1.16 |
| Rob Dauster, SI | $1.11 |
| Jeff Borzello, CBS | $1.08 |
| “Chalk” (higher seed) | $1.02 |
| Dan Wetzel, Yahoo | $0.98 |
| Matt Norlander, CBS | $0.96 |
| Colin Cowherd, ESPN | $0.96 |
| Joe Lunardi, ESPN | $0.95 |
| Dennis Dodd, CBS | $0.93 |
| Luke Winn, SI | $0.92 |
| Vegas Favorites | $0.92 |
| Jerry Palm, CBS | $0.91 |
| Andy Staples, SI | $0.90 |
| Pat Forde, Yahoo | $0.88 |
| Gary Parrish, CBS | $0.87 |
| Dick Vitale, ESPN | $0.85 |
| Greg Doyel, CBS | $0.84 |
| Andy Glockner, SI | $0.84 |
| Andy Katz, ESPN | $0.80 |
| Stewart Mandel, SI | $0.78 |
| Seth Davis, SI | $0.77 |
| Brad Evans, Yahoo | $0.74 |
| LeBron James | $0.73 |
| Mike Huguenin, Yahoo | $0.71 |
Jeff Goodman of CBS took first place, thanks to his picks of both Xavier and Florida to make the Sweet 16. It was a tough first weekend for the pundits overall, however, with only 7 of the 26 returning a $1+ yield . Vegas, the best proxy for crowd consensus, significantly underperformed “Chalk” because it incorrectly predicted lower seeds such as UConn and Alabama to win their first round matchups and Wichita State to make the Sweet 16. Meanwhile, Obama and Lachey did quite well; King James did not.
ESPN’s Jay Bilas had the most accurate bracket based on the standard hit rate approach (# of correct calls divided by # of total calls).
Hit Rate
| Jay Bilas, ESPN | 75.0% |
| Barack Obama | 72.9% |
| Jeff Goodman, CBS | 70.8% |
| Peter Tiernan, CBS | 70.8% |
| Nick Lachey | 70.8% |
| “Chalk” (higher seed) | 68.8% |
| Rob Dauster, SI | 66.7% |
| Dan Wetzel, Yahoo | 66.7% |
| Jeff Borzello, CBS | 64.6% |
| Colin Cowherd, ESPN | 64.6% |
| Vegas Favorites | 64.6% |
| Joe Lunardi, ESPN | 62.5% |
| Luke Winn, SI | 60.4% |
| Jerry Palm, CBS | 60.4% |
| Andy Staples, SI | 60.4% |
| Gary Parrish, CBS | 60.4% |
| Dick Vitale, ESPN | 60.4% |
| Matt Norlander, CBS | 58.3% |
| Dennis Dodd, CBS | 58.3% |
| Andy Glockner, SI | 58.3% |
| Pat Forde, Yahoo | 56.3% |
| Andy Katz, ESPN | 56.3% |
| Stewart Mandel, SI | 56.3% |
| Greg Doyel, CBS | 54.2% |
| Seth Davis, SI | 54.2% |
| Brad Evans, Yahoo | 52.1% |
| LeBron James | 52.1% |
| Mike Huguenin, Yahoo | 52.1% |
For a breakdown of these two metrics by round, click here.
We will update these rankings following the Championship game, using Final Four and Championship futures odds.
Since we launched the PunditTracker blog, our inbox has been inundated with the following request: “Track those NFL Draft pundits!” We understand and fully agree with this suggestion. Draft punditry is widely consumed but epitomizes the accountability gap in the prediction industry. Multiple years must pass before outcomes can be determined, and by that time, everyone is focused on the next draft. The old predictions are long forgotten.
We are working on a project to track the historical accuracy of pundit player rankings (a.k.a. Big Boards). It is a complex task, given that there is no standardized rating system for football players (e.g. how do we compare a defensive back to a running back?). We will have more to report on this project in the coming months and welcome any suggestions in the interim.
For now, let’s turn our attention to those ubiquitous mock drafts, whereby pundits predict which players will go to which teams. This is a more layered — and we would argue less interesting — prediction format, since pundits must devalue their own player rankings in favor of their views of each team’s needs and player evaluations.
We have dug up and graded the Round 1 mock drafts of Mel Kiper (ESPN), Todd McShay (ESPN), and Mike Mayock (NFL.com) from 2009 through 2011. We calculated a variability measure based on the difference between when each player was projected to be picked and when he was actually picked. A lower variability score is better.
The results tell a clear story: Todd McShay has issued the most accurate mock drafts for each of the past three years.
| Variability | 2009 | 2010 | 2011 | Average |
| McShay | 8.7 (#1) | 8.8 (#1) | 6.8 (#1) | 8.1 (#1) |
| Kiper | 8.8 (#2) | 10.4 (#3) | 7.7 (#2) | 9.0 (#2) |
| Mayock | 9.6 (#3) | 9.9 (#2) | 9.3 (#3) | 9.6 (#3) |
Here is a breakdown of the major hits and misses for the group:
Hits and Misses
We will revisit this post next month following the 2012 draft.
Following the conclusion of the 84th Annual Academy Awards last night, it seemed only natural to turn our attention to the pundits who make their living predicting which Hollywood stars will come home with the world’s most infamous little gold statuettes. We hope the data below will prove informative, especially for those who spend countless hours trying to win their Oscar Pool with all kinds of techniques that range from picking the winners of the other awards shows to avoiding nominees that have already won. We would submit that the best technique may simply be to follow the pundit who has demonstrated time and again the ability to pick the most winners.
2012 is our first step in cataloguing those records. We chose 21 well-known film pundits, each of whom made predictions for all the Oscar categories.
Here are the rankings by hit rate (# of correct calls divided by # of total calls):
Hit Rate
| Clayton Davis, The Awards Circuit | 83.3% |
| Melena Ryzik, New York Times | 83.3% |
| Kris Tapley, HitFix | 79.2% |
| Peter Knegt, Indiewire | 79.2% |
| Glenn Whipp, The Envelope | 79.2% |
| Vegas favorites | 77.1% |
| Pete Hammond, Deadline.com | 75.0% |
| Steve Pond, TheWrap | 75.0% |
| Sasha Stone, Awards Daily | 75.0% |
| Dave Karger, Entertainment Weekly | 75.0% |
| Anne Thompson, Indiewire | 70.8% |
| Guy Lodge, HitFix | 66.7% |
| Kyle Buchanan, Vulture | 66.7% |
| Scott Feinberg, Hollywood Reporter | 66.7% |
| Kevin Polowy, NextMovie | 66.7% |
| Lou Lumenick, New York Post | 65.2% |
| Jim Slotek, Toronto Sun | 58.3% |
| Rotten Tomatoes | 58.3% |
| Mark Harris, Grantland | 58.3% |
| Richard Roeper, Chicago Sun-Times | 58.3% |
| Erik Childress, Movies.com | 54.2% |
These results are decidedly mediocre. Simply picking the favorites as deemed by Las Vegas oddsmakers (a proxy for crowd consensus) would have yielded a higher hit rate than 16 of our 21 pundits.
As we have discussed in previous posts, we believe the hit rate approach is flawed. All calls are not equal and should not be scored as such. Instead, pundits should receive more credit for predictions that are out-of-consensus. For instance, Christopher Plummer was a virtual lock for Best Supporting Actor, and so that pick should result in less credit than, say, choosing “Hugo” for Cinematography (over the favored “The Tree of Life”). Our “$1 Bet Yield” metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections. Here is how the Oscar pundits stacked up using this approach:
$1 Bet Yield
| Kris Tapley, HitFix | $1.39 |
| Peter Knegt, Indiewire | $1.37 |
| Clayton Davis, The Awards Circuit | $1.35 |
| Melena Ryzik, New York Times | $1.35 |
| Pete Hammond, Deadline.com | $1.22 |
| Glenn Whipp, The Envelope | $1.16 |
| Steve Pond, TheWrap | $1.14 |
| Sasha Stone, Awards Daily | $1.11 |
| Vegas favorites | $1.10 |
| Dave Karger, Entertainment Weekly | $1.09 |
| Anne Thompson, Indiewire | $1.07 |
| Guy Lodge, HitFix | $1.01 |
| Kyle Buchanan, Vulture | $1.01 |
| Lou Lumenick, New York Post | $0.98 |
| Jim Slotek, Toronto Sun | $0.98 |
| Scott Feinberg, Hollywood Reporter | $0.96 |
| Kevin Polowy, NextMovie | $0.95 |
| Rotten Tomatoes | $0.83 |
| Mark Harris, Grantland | $0.76 |
| Richard Roeper, Chicago Sun-Times | $0.73 |
| Erik Childress, Movies.com | $0.63 |
| S.T. VainAirsdale, MovieLine | $0.59 |
Here, the pundits fare a bit better, thanks largely to underdog picks such as “Undefeated” in Documentary and “Hugo” in both Cinematography and Visual Effects. Still, the performance is uninspiring, with less than 40% of the group outperforming Vegas.
Next year, we plan to track the awards predictions earlier in the process. The other award shows (Screen Actors Guild, Golden Globes, etc) tend to make the market more efficient, which in turn makes it more difficult to generate high Bet Yields come Oscar night, particularly in the major categories.
Update (2/27/12): Added one more pundit, Melena Ryzik of the Carpetbagger blog (New York Times).
It should come as no shock that one of the financial pundits we will be tracking is Jim Cramer. Since January 2011, we have been cataloging all the featured stock picks on his show Mad Money. We assume a one-year holding period for each of his picks and consider a “sell” rating to be a recommendation to short the stock. We then compare the average return of his calls to the comparable average return for the S&P 500 index.
Here are the results through February 15 (he made 28 picks from 1/1/11 through 2/15/11):
Average Return
Jim Cramer: -6.4%
S&P 500: 1.5%
Using our “$1 Yield” metric
Jim Cramer: $0.94
S&P 500: $1.01
We will be updating these figures on a monthly basis.
As our readers know by now, we consider the “hit rate” approach typically used to evaluate pundits to be flawed. This approach assumes that all predictions are equal, which is decidedly not the case. When we embarked on addressing the accountability gap in the prediction industry, we knew an alternative scoring system was needed.
Our solution will be to calibrate each prediction for boldness. We will measure this by asking our users how likely they think a given prediction is to occur. If everyone says “unlikely,” then the call is bold, and the pundit, if correct, should receive more credit than he would for a called deemed “likely”. This moment-in-time gauge of consensus opinion will underpin our scoring algorithm.
The trick, of course, is getting people to vote. While we could bank on the appetite of our user base to hold pundits accountable, we have decided to throw in an extra twist: we will score our users as well. When users vote on the boldness of a pundit’s prediction, we have all the data needed to grade them just as we do our pundits (on an opt-in basis, of course). We envision a significant social component to the website, including user rankings and the ability for users to submit their own predictions once they achieve a certain score.
This notion of user scores should not only make the website interactive and engaging but also lend more integrity to our underlying pundit scoring algorithm. In the coming weeks, we will be providing more examples of how the scoring system will work. As always, we welcome any feedback.