In the weeks leading up to the 2013 NFL draft, FiveThirtyEight published No Team Can Beat the Draft. In essence, it shows that there is no evidence that certain teams are better at drafting than others.
This means we can perform the same test at the GM level as well — and, once again, there’s virtually no relationship8 between how well a GM drafts, relative to average, from one year to the next.
The -20 to 20 scale is a measure of a players Approximate Value relative to the average Approximate Value of players selected in their draft slot. This chart plots the relationship between how well a team does one year against how well they did the previous year. If certain teams were notably better at drafting than others, there should be a distinct positive correlation between the plots, with most falling in either the upper right or lower left quadrant. Instead, it’s an unintelligible mess.
Anecdotal evidence exists of certain General Managers hitting on more picks than average. Panthers fans have seen that with Marty Hurney’s draft success in the first round. Chase Stuart points to two other examples in Bill Polian of the Colts and A.J. Smith of the Chargers in the mid-2000s. Both built up reputations as draft geniuses after stretches of magnificent drafting that left both of their teams stacked with talent. That changed drastically starting in 2007. Between 2007 and 2012, the pair made 79 draft selections. Three of those have made a Pro Bowl- Eric Weddle, Melvin Ingram, and Pat McAfee.
The randomness of draft pick success also puts into question the efficacy of sacrificing picks to move up in the draft. Since teams don’t have a reliable way to accurately forecast who the correct pick should be, there is little benefit in moving up in the draft to get “their guy.” Chances are their guy doesn’t have any greater chance of succeeding than others selected in that same pick range.
Look no further than the Carolina Panthers in the 2015 draft as a perfect example of this. They sent the Rams the 57th, 89th, and 201st pick in exchange for the 41st pick. The Panthers used that pick to select Devin Funchess, who has developed into a fine wide receiver. However, if the Panthers stood pat, they could have selected Tyler Lockett with the 57th pick and kept those later picks. They got their guy, but it has yet to prove beneficial to the team.
This chart between the famous draft pick trade value chart compared to the average Approximate Value of each pick shows how teams view their own ability to draft.
If The Chart is an accurate gauge of how teams value each draft slot, then NFL decision-makers place an incredible premium on high draft picks. But the huge disparity between the observed performance of each pick and its apparent market value supports Massey and Thaler’s hypothesis that teams are not being realistic about their own ability to differentiate among prospects.
The extremely high value placed on high picks implies that NFL decision makers trust themselves to make the right selection when they pick near the top of the draft. That has never proven to actually be the case.
Research by TheBigLead’s Jason Lisk (then writing for Pro-Football-Reference) shows that teams with top-five picks in the draft correctly identify the player who goes on to have the best career only 10.3 percent of the time, a success rate that only gets worse as things progress deeper into the draft.
To counter their inability to consistently and reliably make the best selections possible, NFL teams should look to amass as many picks as possible. As per usual, the Patriots have embraced this forward thinking approach. They consistently trade back in the draft while other teams are overvaluing higher picks, and they have a great amount of success to show for it.
So next time you want to argue that Marty Hurney is a good or bad drafter or want to start up a Hurney vs. Gettleman draft pick debate, remember, it’s almost entirely luck. You might as well be arguing which one you think is better at playing roulette.