Conclusions
Over a dinner table once, years ago, a question was asked: "For how many more years will Josh Allen play well for"? Around the table were multiple football fans, each of whom roots for a different team. The Buffalo Bills fan was convinced that Josh Allen would play well (and remain on the team) for almost a decade more. The Chiefs fan and the Patriots fan disagreed and suggested that this was implausible, due to his aggressive playstyle filled with running plays and hard hits. The more of the story was that everyone around the table had an opinion, but all were based entirely on emotions and anecdotal tid bits, not on grounded science.
And so a question was born. Could one do better? Could NFL football players's careers be systematically studied, with mathematics, programming tools, machine learning algorithms, etc, in order to reliably, objectively, and scientifically predict the future of current NFL player's career paths? At play were a few deeper questions: Would there be enough patterns within the historical data to generate fairly accurate predictions? Or perhaps do NFL careers follow rather more random trends, strong subject to chaotic dynamics? And then secondly, even if said patterns exist, could modern machine learning tools dive deeply enough to find those patterns and return to the surface with usable, meaningful insights? This project has been that very deep dive.
Over months of research, thousands of historical NFL careers were examined, countless predictions were tested, and one idea after another was explored in the pursuit of an objective answer. The result was not merely a prediction about one single quarterback, but a much broader understanding of what can—and cannot—be learned from the careers of professional football players using data science.
Throughout the course of the project, many different versions of the predictive engine were crafted and then systematically tested against the same collection of historical NFL careers. Each version represented another attempt to generate predictions a little more accurately than had been done before. Sometimes there was success. Other times failure. With every version of the predictive engine finally complete, the moment has arrived to compare them side by side and discover which ones truly rose above the rest:
The results were fascinating. The very first version, Nearest Neighbor with Mode, performed admirably but was fairly quickly surpassed by its cousin-engine Nearest Neighbor with Median. This fairly simple early prototype proved to be surprisingly difficult to beat, outperforming many of the newer, more-advanced ideas that followed. Some approaches produced modest improvements (Decision Trees, reduced form), while others actually moved the predictions quite a lot farther away from the correct answers. The strongest performers of all were all SVM-based engines, while on the other end of the spectrum, the Neural Network and K-Means versions fell well short of expectations.
While comparing the different prediction engines reveals which approaches performed the best overall, numbers and graphs can only tell part of the story. The real beauty of this finished project is watching it make predictions for real NFL players and comparing those forecasts with the careers that actually unfolded. The following examples showcase the final predictive engines in action. They show the original mode-based version in purple, decision trees version in red, and SVM version in green. They illustrate both the joy of how powerful these predictive machines can be, and also sometimes their limitations or failures.
The rest of quarterback Kurt Warner's career, given the first 7 seasons:
The rest of kicker Dan Bailey's career, given the first 5 seasons:
The rest of Patriots cornerback Malcom Butler's career, given the first 4 seasons:
These three examples were chosen at random and were not cherry-picked for their accuracy. We can see that they demonstrate the relatively "not bad" performance of the predictive models! Player retirement is seen, as was suggested by the earlier graph, to often be accurate to within ± a year or sometimes two. And the career performance of the players, while the various engines often gently disagree, are seldom completely wrong. The predictive engines fairly-well capture the general trajectory of the careers, often predicting both the length of their playing days and the level at which they continued to perform with fairly impressive accuracy.
Code
For any interested in testing the final user-friendly version of the predictive engine out for themselves, the entire complete code is featured below:
The finished system demonstrates that meaningful and surprisingly accurate long-term career predictions are indeed possible. With the project now complete, it is finally time to return to the ten questions posed at the very beginning and discover which ones this research was able to answer.
The 10 Questions
(1) How good are the predictions? What percent of the time did the machine get it right?
This was an area of great success for the project! A full, quantitative assessment methodology was achieved whereby any given prediction of a historical player could be compared directly to the actual future history of that player's career and the gap between the prediction and the actuality could be examined numerically! This was a very exciting breakthrough within the progress of the project.
As for how well the machine was actually able to do the predictions, there was a slight range of performance across the different builds. But all of them were able to predict the player's retirement to within approximately two years, on average, and all of them were able to predict the future of that player's career to within about ±15% or so of how well they actually continued to play. These results demonstrate that historical NFL data contains enough info to make useful real-world predictions about a current player's future career.
(2) For each player position, how can all of these different statistics be combined into one overall grade that fairly reflects how well a player actually performed?
This was another of great success within the project. Rather than focusing on only a specific handful of statistics, the choice was made to include all available statistics into the project's mechanisms. Every single statistic tells a story, and those stories all put together are the history of what that player did that year. None of it should go to waste. To explain the process, let's start with quarterbacks as an example. Bigger, more meaningful stats like how many touchdowns they got and their QB rating were scaled up (multipled by a coefficient). Littler stats, like how many times they got sacked, which might be more of a reflection on their offensive line than themselves, got divided down by a coefficient. All of these were then added up, normalized, and compared against other QBs of the same season to give a percentile score.
Similarly for kickers, accuracy on short and medium kicks got scaled up. Their number of kicks blocked got scaled down.
Once the grading system had been developed, it was carefully compared against well-respected historical rankings of the league's best players from past seasons. The resulting grades closely matched those expert evaluations, providing confidence that the system was producing fair and realistic assessments of player performance!
(3) What should be done with missing or corrupted data?
This question was addressed in two different ways. The first decision was to limit the project primarily to the modern era of NFL history. While the statistical records from the 1940s are fascinating from a historical perspective, they proved to often be too incomplete and too inconsistent to be reliably incorporated into a modern prediction engine. Furthermore, while I respect endlessly the early work by some of the historical greats of the sport, they did compete in a vastly different era of football, making them less useful for predicting the future careers of today's athletes.
A second challenge emerged, quite unexpected, from within the modern data itself. A surprisingly large quantity of the statistics recorded from the 1990 season were found to be either missing or corrupted. This was true across all players and all player positions... very odd indeed. This project's author still has great curiosity as to what was different that specific year to have caused a kind of "statistics blackout", if you will, and resulted in such suddenly worse quality data just for that one specific year. There is probably a story behind it. Rather than having to discard all player careers who played in 1990, and thus have one corrupted season, various method were employed to extrapolate and fill in those missing values by examining the surrounding seasons of players with similar careers. While no reconstruction can ever be perfect, these techniques allowed the affected data to remain useful within the project.
(4) How to fix the tendency of the machine to have bias towards predicting early retirement?
This was another of the strengths of the project. The original mode-based version of the prediction engine had a strong tendency to expect players to retire earlier than they actually did. As a result, many otherwise accurate career predictions were getting cut short prematurely. The project found that this problem stemmed from trying to answer two very different questions at the same time: For how long will the player continue playing? and also how well will the player perform during those years?
The solution proposed in the Introduction ultimately worked extremely well. Rather than attempting to answer both questions simultaneously, the final prediction engine first estimated when they player would retire. Only after making that decision did it predict how well the player would perform during the remaining seasons of his career. Separating these two decisions dramatically reduced the tendency to predict early retirement and allowed for the creation of far more nuanced, subtle, and accurate career forecasts. This was a big leap in progress!
(5) How can we use machine learning to make the predictive engine more and more precise and accurate?
This question ultimately became the heart of the entire project, and the answer was an enthusiastic yes! Throughout the semester, many completely different versions of the prediction engine were designed and tested, each built upon a different machine learning technique. Every new version represented a fresh attempt to improve upon the strengths of the one before it. Some approaches produced only modest improvements, some performed quite-surprisingly worse than expected, and a few demonstrated dramatic gains in predictive accuracy.
The whole time during the design and construction of each new version, it was deeply exciting wondering how well it would perform. It was legitimately so much fun finally pressing "go" on each new version, and seeing the final accuracy output. Based on the leap in quantitative analysis from question (1) above, it was possible to objectively see which engines worked well and which were flops.
But the core takeaway was that by the end of the project, it became clear that indeed machine learning was capable of producing prediction engines that were substantially more accurate than the original design.
(6) Will there be a trade-off between accurately predicting player retirement and accurately predicting player performance?
Surprisingly, very little evidence of such a trade-off was ultimately found. At the beginning of the project, it seemed entirely possible that one prediction engine might become exceptionally good at predicting retirement while another, very different version, might supersede it at forecasting future player performance. Instead, the opposite pattern generally emerged. Prediction engines that excelled at one task almost always excelled at the other as well. There was extremely solid correlation between effectiveness at the two types of prediction. This was an encouraging discovery, almost a best-case scenario if you will, as it meant that improving out aspect of the predictive engine did not have to come at the expense of the other.
(7) Which version of the predictive engine will prove to be the best overall?
As seen in the earlier plot of all their different performances, one version of the machine was the clear favorite. While it had been expected to have two different prizes — one for best retirement predictor, one for best player performance predictor — one version of the machine took the double crown! The SVM Radial predictive engine earned the title as undisputed champion of the entire project.
(8) Will the later, more advanced machine learning techniques necessarily produce better results than the earlier, simpler ones?
The answer turned out to be a resounding no. One of the biggest surprises of the entire project was discovering that newer and more sophisticated predictive engines often performed worse than the much simpler versions that had come before them. Time and again, exciting new ideas were developed with the hope that they would dramatically improve the predictions, only to discover through careful testing that they actually made the results less accurate, sometimes by quite a large margin!
On the other hand, the single strongest prediction engine of the entire project was also one of the most sophisticated. In the end, the lesson was not that newer ideas are always better, but rather that every new idea must earn its place through objective testing. Sometimes a more advanced approach truly does represent a breakthrough—but just as often, simplicity wins the day.
(9) Where should the project go from here?
There are many areas of improvement yet to be seen in this project's future: First, much of the project was completed in a slightly rushed manner, under deep time constraints. It would be nice to go back through each and every corner of the project and polish everything, check each code for bugs, and make sure all the writeups are as polished and perfect as can be.
Second, the project right now has rigid, set datacubes of historical info. It would be preferable to get the project to be able to quickly, easily, and in an automated fashion apply each next-year's new NFL data to the datacubes! This should all be automated, even the normalizations and data cleaning. That would be be inspiring and would let the project have renewed life with each new season, as the new data comes out.
Third, right now the predictive engine can only operate on either historical players or current players, given all their complete up-to-date data. But it would be lovely to add to this the ability to let the user predict what the second half of a current player's career might look like, based on only some set first number of years of training data from their yet-unfinished career! This might be very insightful and worthy of examination. This is especially relevant as people have more of an emotional connection to their team's current players, so it might give the project more flair.
Lastly, and most profoundly, it would be wonderful explore more machine learning tools! Most notably, it would be nice to add a whole additional page on XGBoost. It sounds both fascinating and highly relevant for this project's objectives. It would be great to explore a few of the more cutting-edge machine learning tools, like XGBoost and Random Forests, and see how well those versions of the predictive engine perform.
(10) Lastly, returning all the way to where the project originally started with a question over a dinner table —— For how many more years will Josh Allen play well for?
Over a year ago the original version of the predictive engine was designed and built, and finally gave an answer to that question:
Josh Allen is expected to continue playing at a high level for approximately three more seasons, followed by one season of dramatically lower performance before ultimately retiring from professional football. If he wants to win a Super Bowl, he does not have much time left!
This made the author of this report, as a Patriots fan, very happy. But a year has passed, extensive more work has been done on designing better and better NFL predictive modeling machines. Let's see what the new-and-improved better engines suggest as their predictions:
This result is stunning and downright hilarious. After almost a year of having the early stages of final concrete proof that the Buffalo Bills fan in the house was quite incorrect with his assessment, the new-and-improved later generation versions of the predictive engine both agree in perfect union that Josh Allen is now, and will remain, one of the greats. They each predict him to play at the absolute top of the world for at least another seven years. What a truly fascinating reversal. Perhaps the Buffalo Bills fan, more than two years after the original dinner table discussion, has finally been vindicated and shown to be correct after all.
The question of how well Josh Allen will play going forward, however, was never really the true goal of the project. The project was a means by which to explore data science, mathematical modeling, and machine learning. The question about Josh Allen provided the spark that inspired an entire epic journey through data science, and led to a deep and fulfilling research expedition. It has been a fascinating journey, and this author excitedly awaits to see where the project will next venture to.