As we continue to expand both the number of pundits and types of pundits we track, we need your help. We are looking for volunteer Moderators and Trackers.
Moderators will take “ownership” of specific categories, helping to shape creative direction. For instance, we are considering tracking Economists in Finance and Pollsters in Politics, but these categories have gray areas that create scoring challenges. For instance, both economists and pollsters frequently revise their predictions/estimates, so how should we go about tracking them? And how should we think about margin of error? Rather than imposing our own views, we think it’s best to give authority to our users, who are more knowledgeable about these specialties than we are.
As far as Trackers, as we scale up the number of pundits we track, it becomes more difficult to catch all the predictions that are being made. As such, we are looking for Trackers who will help us screen the daily newsflow for new predictions in a given category or even for a specific pundit or television show.
Here are some of the categories for which we are looking for Moderators and Trackers, although if you have suggestions for new categories, we are all ears:
Finance: Economists (e.g. GDP, inflation), Stock Market (e.g. stock analysts, annual S&P predictions), General
Politics: Pollsters, General
Sports: NFL, MLB, NBA, NCAA Football, NCAA Basketball, Other (e.g. Hockey, Boxing, Horse Racing, Tennis, UFC/MMA)
Entertainment: General (e.g. Oscars, Emmys)
Technology: General
Use the “Whom should we track?” button on the bottom-right of the page to contact us and we will get back to you promptly. Thanks!
The 2013 NFL Draft was widely viewed to be more unpredictable than most. After grading the mock drafts of Mel Kiper (ESPN), Todd McShay (ESPN), and Mike Mayock (NFL.com), we can confirm that those views proved correct: all three pundits had a rough year. Todd McShay, who was #1 in mock draft accuracy in each of the past four years, had a particularly woeful showing.
Let’s start with our simple variability measure, which is based on the difference between when each player was projected to be picked and when he was actually picked. A lower variability score is better:
Variability 2008 2009 2010 2011 2012 2013 Average
Mayock 7.3 9.6 9.9 9.3 7.9 10.7 9.1
Kiper 10.1 8.8 10.4 7.7 8.5 11.7 9.6
McShay 10.7 8.7 8.8 6.8 7.0 21.5 10.6
The average variability for the three pundits from 2008-12 was 8.8, which underscores just how poor their 2013 performance (14.7 average) was. Big misses for all three included Sharrif Floyd going #3 to Oakland (he went #23 to Minnesota) and Eddie Lacy going late in the first round (actually went late second round). McShay and Mayock both whiffed on Geno Smith, picking him to go #6 to Cleveland (actual: #39 to Jets).
The worst projection of all, however, was McShay pegging Ryan Nassib going #7 to Buffalo (actual: #110 to the Giants). This was by the far the worst first-round projection since we started tracking the mock drafts in 2008.
Here are all the major hits and misses:
Biggest Hits and Misses (2008-2013)
The variability metric can be significantly impacted by “outliers” such as Nassib, so we have calculated a host of alternative metrics, including:
- Capped Variability (lower = better): caps the maximum penalty of any pick at one round — for instance, McShay’s Nassib pick would be graded as 32 spots off rather than 103 spots
- Median Error (lower = better)
- Hit Rate (higher = better): pure percentage of how many picks were correct
Here are the results for these three metrics from 2008-2013.
Capped Variability 2008 2009 2010 2011 2012 2013 Average
Mayock 7.3 9.6 9.3 9.3 7.9 10.3 8.9
McShay 10.7 8.7 8.7 6.8 7.0 12.3 9.1
Kiper 10.1 8.2 9.5 7.7 8.5 10.9 9.3
Median Error 2008 2009 2010 2011 2012 2013 Average
Mayock 2.0 4.0 3.0 3.0 3.5 3.5 3.2
Kiper 3.0 3.5 3.0 4.0 3.0 3.5 3.4
McShay 4.0 4.0 3.0 3.5 3.0 5.0 3.8
Hit Rate 2008 2009 2010 2011 2012 2013 Average
Mayock 35% 28% 22% 28% 19% 28% 27%
Kiper 29% 25% 31% 28% 22% 25% 27%
McShay 16% 25% 34% 25% 16% 22% 23%
Will Tavon Austin win Offensive Rookie of the Year? Peyton Manning for MVP? Make your Predictions now on PunditTracker while the voting is still open — click here to submit your picks (use the VOTE NOW buttons).
After announcing the idea of PunditTracker early in 2012, we solicited suggestions from our readers on which pundits to track. The most requested financial pundit, by far, was Jim Cramer. In fact, during our interview with NPR last March, host Bob Garfield ended the discussion with the not-so-subtle request: “Two words: Jim Cramer”.
We are making Cramer the first in our PT Focus series, in which we take an in-depth look at a pundit’s track record. For those who are unaware, Cramer is the host of the popular show Mad Money on CNBC, providing general financial advice as well as specific stock recommendations.
Cramer currently has an F grade on PunditTracker.com. Let’s walk through how we arrive at that score.
First, here are the parameters by which we evaluate Cramer’s stock recommendations:
- We started tracking his picks on January 1, 2011.
- We score only his “Featured” picks on Mad Money and not those made in other segments such as the “Lightning Round.” Our reasoning is that the Featured picks are unsolicited recommendations for which Cramer has presumably done more research than for picks given as live responses to viewer questions.
- We measure the performance of his picks relative to that of the S&P 500 index over the corresponding period.
- We assume a three-month holding period, unless Cramer reverses his stance on a given name (e.g. says Buy XYZ and then says Sell XYZ within the three months), in which case we “close out” the original recommendation. This holding period is based on the idea that Cramer tends to revisit his picks each quarter. (Note: We also have calculated performance using a six-month holding period; we are happy to provide the data if there is interest).
- The baseline stock price is the opening price two days after the recommendation is made, in order to account for any day-one “Cramer bump” effect.
- The hit rate is the percentage of Cramer’s picks that outperform the index. We equate sell ratings to short recommendations (i.e. they are scored as correct if the stock underperforms the S&P).
With the parameters out of the way, let’s now delve into the details of Cramer’s performance.
Given our assumed three-month holding period, we have now graded two years worth of Cramer’s picks: those made from January 2011 through December 2012. That amounts to 552 calls overall, of which 254 outperformed the index (46% hit rate).
On average, Cramer’s picks returned -0.08% versus the 1.35% S&P 500 return over the corresponding period. That amounts to 142 basis points of quarterly underperformance, or 568 basis points on an annualized basis, which amounts to an F grade in our grading system. (We award an A for 500+ basis points of annual equity outperformance and an F for 500+ basis points of underperformance).
Let’s now break down his results by year and by quarter:
| 2011 | 2012 | Total | |
|---|---|---|---|
| Cramer's Picks (three-month holding period) | -1.6% | 1.1% | -0.1% |
| S&P 500 | 0.5% | 2.0% | 1.4% |
| Relative performance (basis points) | -207 | -89 | -142 |
| # picks | 247 | 305 | 552 |
| Hit Rate (% that outperformed) | 48% | 44% | 46% |
| PunditTracker Grade | F | D | F |
| 1Q11 | 2Q11 | 3Q11 | 4Q11 | 1Q12 | 2Q12 | 3Q12 | 4Q12 | |
|---|---|---|---|---|---|---|---|---|
| Cramer | -1.1% | -8.3% | -5.3% | 6.0% | 1.9% | 2.5% | -3.1% | 2.1% |
| S&P 500 | 1.2% | -4.8% | 0.0% | 4.7% | 1.6% | 2.8% | 0.1% | 3.1% |
| Relative | -234 | -347 | -538 | 133 | 29 | -35 | -313 | -101 |
| # picks | 59 | 63 | 48 | 77 | 86 | 81 | 61 | 77 |
| Hit Rate | 39% | 46% | 46% | 58% | 45% | 43% | 36% | 51% |
| PT Grade | F | F | F | A | C+ | C- | F | D- |
Here were Cramer’s ten best and worst picks during the two years:
10 Best Picks
| Date | Pick | S&P | Relative | |
|---|---|---|---|---|
| Sell RSH | 12/22/11 | 33.3% | -12.0% | 45.3% |
| Sell RIMM | 05/02/11 | 50.0% | 7.5% | 42.5% |
| Buy TRNX | 03/18/11 | 40.4% | -1.5% | 41.9% |
| Sell VRA | 03/02/12 | 45.5% | 6.3% | 39.2% |
| Sell SVU | 07/11/12 | 31.9% | -7.0% | 38.9% |
| Sell MDRX | 03/15/12 | 39.2% | 4.4% | 34.8% |
| Sell ZIP | 06/11/12 | 24.7% | -8.3% | 32.9% |
| Sell FSLR | 10/13/11 | 27.5% | -5.3% | 32.7% |
| Sell TUMI | 04/23/12 | 35.1% | 2.5% | 32.7% |
| Sell NOK | 05/02/11 | 39.7% | 7.5% | 32.2% |
10 Worst Picks
| Date | Pick | S&P | Relative | |
|---|---|---|---|---|
| Sell VRUS | 06/09/11 | -130.0% | 9.2% | -139.2% |
| Buy NFLX | 07/26/11 | -70.0% | -4.8% | -65.1% |
| Sell Z | 11/29/12 | -64.3% | -7.2% | -57.2% |
| Buy PACB | 06/24/11 | -65.8% | -9.2% | -56.6% |
| Sell HPQ | 11/20/12 | -59.9% | -9.0% | -50.9% |
| Sell ZIP | 11/19/12 | -58.6% | -10.3% | -48.3% |
| Sell FB | 09/04/12 | -47.9% | -0.4% | -47.5% |
| Buy QCOR | 09/07/12 | -47.8% | -0.7% | -47.0% |
| Buy OCLR | 02/22/11 | -44.7% | 1.0% | -45.7% |
| Sell TRLA | 11/29/12 | -52.4% | -7.2% | -45.3% |
Cramer underperformed the index on both his Buy and Sell recommendations, although his sells did even worse, with the stocks rising 4.9% on average versus 3.2% for the S&P.
Buy Sell (Short)
# of Picks 494 122
Cramer 1.1% -4.9%
S&P 2.4% -3.2%
Relative -135 -173
We will provide quarterly updates on Cramer’s performance going forward.
Can you pick stocks better than Jim Cramer? For those new to PunditTracker, we allow users to compete with pundits. Click here to visit our Finance page and start making predictions on various financial events, including Apple’s quarterly performance tomorrow (using the “Vote Now” buttons).
With the playoffs tipping off tomorrow, let’s revisit how the pundits did with their regular season predictions. We are tracking the predictions of 35 ESPN pundits across 11 categories: 6 division winners, 2 conference winners, overall champion, MVP, and Rookie of the Year. In this post, we will analyze the 6 division winners and MVP/ROY, assuming that LeBron James and Damian Lillard win the player awards, respectively.
The table is ranked by Yield, although you can sort by # correct as well. As a reminder, the Yield metric calibrates predictions for boldness by measuring the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections.
Pundit Atl Ctl SE NW SW Pac RoY MVP # Yield
Kevin Arnovitz BOS IND MIA OKC SA LAL Lillard James 6 $1.47
Nick Friedell BOS IND MIA OKC SA LAL Lillard James 6 $1.47
Dave McMenamin BKN IND MIA OKC MEM LAL Lillard James 5 $1.27
Israel Gutierrez BOS IND MIA OKC SA LAC Davis Paul 5 $1.21
Brian Windhorst BOS IND MIA OKC SA LAL Lillard Durant 5 $1.12
Beckley Mason BOS IND MIA OKC SA LAL Lillard Durant 5 $1.12
Mike Mazzeo BOS IND MIA OKC SA LAL Lillard Durant 5 $1.12
David Thorpe PHI CHI MIA DEN SA LAL Lillard James 4 $1.11
Marc Stein BOS IND MIA OKC SA LAL Davis James 5 $1.04
Chris Palmer BOS IND MIA OKC SA LAL Davis James 5 $1.04
Larry Coon BOS IND MIA OKC SA LAL Davis James 5 $1.04
Royce Webb BOS IND MIA OKC SA LAL Davis James 5 $1.04
Tom Haberstroh BOS IND MIA OKC SA LAL Davis James 5 $1.04
Maurice Brooks BOS IND MIA OKC SA LAL Davis James 5 $1.04
Arash Markazi BOS IND MIA OKC SA LAL Davis James 5 $1.04
Chris Forsberg BOS IND MIA OKC SA LAL Davis James 5 $1.04
Ian Begley BOS IND MIA OKC SA LAL Davis James 5 $1.04
Vegas Favorites BOS IND MIA OKC SA LAL Davis James 5 $1.04
Scoop Jackson BOS IND MIA OKC MEM LAC Drummond Paul 4 $1.02
Justin Verrier BOS IND MIA DEN SA LAL Lillard Durant 4 $0.98
R. Shelburne BOS IND MIA OKC MEM LAL Lillard Paul 4 $0.93
Tim Legler BOS CHI MIA OKC SA LAL Lillard Durant 4 $0.90
Keith Lipscomb BOS CHI MIA OKC SA LAL Lillard Durant 4 $0.90
John Hollinger BOS IND MIA DEN SA LAL Davis James 4 $0.89
Chad Ford BOS IND MIA DEN SA LAL Davis James 4 $0.89
Adena Andrews BOS IND MIA OKC MEM LAL Davis James 4 $0.84
Jackie MacMullan BOS CHI MIA OKC SA LAL Davis James 4 $0.82
Tim MacMahon BOS CHI MIA OKC SA LAL Davis James 4 $0.82
Chris Ramsay BOS IND MIA DEN MEM LAL Davis James 3 $0.70
J.A. Adande BOS IND MIA OKC SA LAL Davis Durant 4 $0.69
Chris Broussard BOS IND MIA OKC SA LAL Davis Durant 4 $0.69
Michael Wallace BOS IND MIA OKC SA LAL Kidd-Gilchrist Paul 4 $0.69
Henry Abbott PHI IND MIA OKC SA LAL Drummond Durant 4 $0.69
Jack Ramsay PHI IND MIA OKC SA LAL Sullinger Wade 4 $0.69
Bruce Bowen BKN IND MIA OKC SA LAL Davis Durant 4 $0.69
Adry Torres BKN IND MIA OKC MEM LAL Davis Durant 3 $0.50
Here is a breakdown of the percentage of pundits that got each category correct, alongside the Vegas implied odds for each winner before the season began. For instance, not one of the pundits picked the Knicks to win the Atlantic while only two picked the Clippers to win the Pacific, even though Vegas gave each of them 24% odds.
Category % of Pundits Vegas Odds
Knicks Win Atlantic 0% 24%
Pacers Win Central 86% 57%
Heat Win Southeast 100% 95%
Thunder Win Northwest 86% 86%
Spurs Win Southwest 83% 64%
Clippers Win Pacific 6% 24%
LeBron James Wins MVP 54% 36%
Damian Lillard Wins ROY 31% 29%
Finally, let’s look at the Finals and Championship predictions for the group.
Pundit Yield
(Reg Season 2012)East West Champion
Kevin Arnovitz $1.47 MIA OKC MIA
Nick Friedell $1.47 MIA LAL MIA
Dave McMenamin $1.27 MIA LAL MIA
Israel Gutierrez $1.21 MIA LAL MIA
Beckley Mason $1.12 MIA OKC OKC
Brian Windhorst $1.12 MIA OKC MIA
Mike Mazzeo $1.12 MIA LAL MIA
David Thorpe $1.11 MIA LAL MIA
Arash Markazi $1.04 MIA LAL LAL
Chris Forsberg $1.04 BOS LAL LAL
Chris Palmer $1.04 MIA LAL MIA
Ian Begley $1.04 MIA LAL MIA
Larry Coon $1.04 MIA LAL LAL
Marc Stein $1.04 MIA LAL MIA
Maurice Brooks $1.04 MIA LAL LAL
Royce Webb $1.04 MIA OKC MIA
Tom Haberstroh $1.04 MIA LAL MIA
Scoop Jackson $1.02 MIA LAL MIA
Justin Verrier $0.98 MIA OKC MIA
R. Shelburne $0.93 MIA LAL MIA
Keith Lipscomb $0.90 MIA OKC MIA
Tim Legler $0.90 MIA OKC MIA
Chad Ford $0.89 MIA LAL MIA
John Hollinger $0.89 MIA SA MIA
Adena Andrews $0.84 MIA LAL LAL
Jackie MacMullan $0.82 MIA LAL LAL
Tim MacMahon $0.82 MIA LAL MIA
Chris Ramsay $0.70 MIA LAL LAL
Bruce Bowen $0.69 MIA OKC MIA
Chris Broussard $0.69 MIA LAL MIA
Henry Abbott $0.69 MIA LAL MIA
J.A. Adande $0.69 MIA LAL MIA
Jack Ramsay $0.69 MIA OKC MIA
Michael Wallace $0.69 MIA LAL MIA
Adry Torres $0.50 MIA LAL LAL
Can you do better than these so-called experts? There are still a few hours left to make your predictions for the Round 1 of the NBA Playoffs. Will the Knicks beat the Celtics and the Clippers defeat the Grizzlies, as Kevin Arnovitz is predicting? Make your predictions by visiting our Sports page and using the “Vote Now” buttons.
With the baseball season underway, here is an overview of what the MLB pundits are predicting this year. As we have discussed, despite the concerns about payroll disparity, baseball has proved the toughest sport to predict in recent years. If you had bet $1 on each of the baseball pundit picks over the past three seasons, you would have lost 44% of your money. This compares to an 8% loss for NFL pundits and an 18% gain for NBA pundits.
Will 2013 be be the year in which the MLB pundits turn it around? We will be tracking 63 so-called experts from ESPN, Sports Illustrated, CBS, Yahoo, and Fox. Here are their aggregate picks across nine categories: six Division Winners, two Pennants, and World Series (the weird formatting is for sorting purposes).
AL(E) AL(C) AL(W) NL(E) NL(C) NL(W) AL
NL W. Series
30: TOR 62: DET 37: LAA 59: WAS 52: CIN 32: SF 39: DET 44: WAS 28: WAS
29: TB 01: KC 16: TEX 04: ATL 11: STL 23: LAD 12: TB 10: CIN 23: DET
03: BAL 00: CHW 10: OAK 00: MIA 00: CHC 07: ARI 06: TOR 05: LAD 04: CIN
01: NYY 00: CLE 00: SEA 00: NYM 00: HOU 01: SD 06: LAA 04: ATL 03: LAA
00: BOS 00: MIN 00: PHI 00: PIT 00: COL 03: TB
01: TOR
01: LAD
There is still time for you to make your own picks for all these categories — and compete with the experts to win prizes — on PunditTracker’s Sports page (simply use the VOTE NOW buttons).
As the table shows, Nationals vs. Tigers is the en vogue choice for the World Series. The tightest races, according to the pundits, will be the AL East and NL West, with the group roughly split on whether the Blue Jays or Rays will take the former and the Giants or Dodgers the latter. Interestingly, the Rays and Dodgers are the more popular choices for the AL and NL Pennant, respectively. Some of the more contrarian picks we are tracking include: Dodgers to win the World Series (Andrew Marchand, ESPN), Padres to win the NL West (Joe Sheehan, SI), Royals to win the AL Central (Jon Morosi, Fox), and Yankees to win the AL East (Mark Simon, ESPN).
Here are the predictions for the individual player awards (ESPN did not predict these):
AL MVP NL MVP AL Cy Young NL Cy Young AL ROY NL ROY
08: Trout 11: Votto 10: Verlander 09: Strasburg 07: Myers 06: Gyorko
04: Longoria 06: Harper 04: Darvish 07: Kershaw 06: Hicks 04: Teheran
03: Cabrera 01: Upton 02: Price 01: Wainwright 03: Bundy 02: Taveras
02: Bautista 01: Posey 02: Hernandez 01: Hamels 02: Profar 01: Miller
01: Pujols 01: Braun 01: Weaver 01: Cain 01: Maurer 01: Eaton
01: Fielder 01: Johnson 01: Greinke 01: Rondo 01: Cole
01: Cano 01: Ryu
01: Wong
01: D'Arnaud
01: Wheeler
Finally, let’s break out the individual predictions of the pundits we have tracked since 2009 (four seasons of predictions). They are ranked by our Yield metric, which measures the average payout (using Vegas odds) had you placed $1 bets on each of the pundit’s selections.
Yield
09-12AL
EastAL
CentAL
WestNL
EastNL
CentNL
WestAL NL World
Series
Jim Caple, ESPN $1.60 TOR DET LAA ATL CIN SF DET CIN DET
Peter Pascarelli, ESPN $1.10 TB DET LAA ATL CIN SF DET CIN DET
Jayson Stark, ESPN $1.07 TOR DET LAA WAS CIN LAD DET WAS WAS
Pedro Gomez, ESPN $0.86 TB DET OAK ATL CIN SF TB CIN CIN
Tom Verducci, SI $0.84 TB DET LAA WAS CIN SF TB WAS WAS
Vegas Favorites $0.82 TOR DET LAA WAS CIN LAD TOR WAS WAS
Albert Chen, SI $0.81 TB DET TEX WAS CIN ARI DET CIN DET
Joe Lemire, SI $0.78 TB DET TEX WAS STL SF TB WAS WAS
Buster Olney, ESPN $0.72 BAL DET OAK WAS CIN ARI DET WAS WAS
Ben Reiter, SI $0.68 TB DET OAK WAS CIN LAD TB LAD TB
Eric Karabell, ESPN $0.57 TB DET LAA WAS STL SF TB WAS TB
Jerry Crasnick, ESPN $0.37 TOR DET LAA WAS CIN LAD DET CIN CIN
To see how all the pundits we track fared last year, see our 2012 MLB recap post.
We have previously analyzed how the NFL Draft Pundits (e.g. Kiper, McShay, Mayock) have done with their mock drafts (see here). We would now like to turn our attention to what we view to be a more interesting draft endeavor: figuring out how the pundits’ player rankings actually turn out. In other words, when Mel Kiper ranks his Top 5 QBs for this year’s draft, how accurate will those rankings be five years down the road?
We are actively discussing many different approaches to evaluate and compare players but we’d like to tap into the knowledge of our user base. If you are interested in this subject, please send us a note (just use the “Who should we track” button at the bottom of the page) and we will follow up with you. Thanks!
