Evaluation Methodolgy
It was time to finally invent a way to give a quantitative evaluation number to exactly how accurate are the predictions the machine generates. Before discussing that, it might be wise to first take a look at how the machine, as first designed, works and operates.
For the moment, attention will be restricted to just historical NFL players, whose careers are completely finished. This is the training set for the machine to get better and better with, and this is the data set that will be used to evaluate the machine's accuracy. The machine takes in a player name and a fixed number of years of career data the user seeks to offer as given data. It takes the full statistics of those given years of that player's career (passing yards, interceptions, touchdowns, etc) and compares them to all other players who ever played the game. It does so by comparing those normalized stats using the standard vector-distance Euclidean metric. It finds the closest 30 players and gathers their full careers. Most worthy of note, it gathers the rest of those players's careers, possibly including many seasons after the given data's number of years.
Those 30 players's careers are converted into letter grades A, B, C, D, F, and 'retired', based on how well they performed, and collected into associated letter-grade bins. The data mining machine uses 'mode' as the statistic by which to generate its predictions. Whichever is the most common one of those bins across each of the next seasons is the letter grade that the machine gives for its prediction. A visual example of this in action is:
This 'mode' based binning style of prediction is then converted back into a numerical score using the conversion {A -> 0.9, B -> 0.7, C -> 0.5, D -> 0.3, F -> 0.1} and finally compared back to the actual years that the player played for.
Two scores result. How far off was the prediction for number of years yet to play before retirement? And of those years, that overlap both in the prediction and in the rest of that historical player's actual career, how far off were the predictions? These two scores will be called the retirement error and the prediction error.
The initial concept was to test this across the full set of all historical NFL players, across all valid given numbers of years (3 through 10). This sounded good at first, but upon actually running the experiment, it turned out to wildly skew the entire project towards the much-larger number of players at the bottom, in terms of NFL careers. Many many more players played badly and briefly for just a few seasons and then retired. The whole project was steering towards becoming all about how well could the machine predict those kind of short-lived career-backups's careers. It was entirely possible, then, that the ideal prediction would just be to have all players play one additional season at an F-level and then promptly retire. This did not seem to be in the spirit of the project at all, as the original core question was "For how much longer will Josh Allen continue to play well for?". A new method was needed. After a good deal of consideration, the following new method was invented.
First the machine will generate a random number from 3 through 10. This will represent the number of seasons of given info that the machine will have offered to it. Only then the machine will randomly select a player who played for at least that long. This means the project will be as evenly focused on players who played only 3 seasons as those who played for 10+. This feels much more in spirit with the goals of the project.
The evaluation metric will be to randomly choose 10000 numbers from 3 through 10, select 10000 random players (with replacement) whose careers were at least long enough, perform 10000 tests, and arrive at the mean retirement error and the mean prediction error. Sometimes the machine will run genuine randomized tests. Sometimes the machine use a set random number seed, for reproducibility.
The first grand test was performed, and this was the result:
Original Version
This is now the baseline to which all future improvements of the project will be compared.
The next big area of improvement that was desired was to try was to eliminate the use of the binning / mode-based prediction style of the original machine. It was desired to somehow use mean to generate the predictions, instead of mode. This seemed more professional and scientific. The next iteration of the machine created worked in the following way:
Collect the 30 closest players. Gather the median number of future years they played past the number of given years. If that value is zero, then simply predict the test player will retire immediately. If not, suppose the median number of future seasons is 2. For each of those 2 more seasons, prune from the 30 any players who did not play at least that long, then take the mean of the percentile score for the remaining non-pruned players. This new and improved version of the machine had the following results:
Mean, not Mode
We see a significant improvement upon the original design! Mean retirement error dropped by 0.1131, while mean prediction error fell by 0.0030. Many variations on this were tested, including finding the nearest 50 players, the nearest 20, the nearest 10... Another variation involved evaluating the career percentiles only of players who retired at a similar time to the expected median retirement years prediction... but none of these later improvements gave statistically significant improvements. That might be an area of further investigation.
One more experiment was run, however, that did in fact have promising results! It was worth testing what would happen if the machine used the median future predictions for the career percentiles instead of the mean. The results of that change were:
Neither Mode Nor Mean, But Median
There was an improvement on the mean prediction error of 0.0005. A small change, but still a change in a good constructive direction.
That also means that this is the best version of the machine to date! This version is what the project will now try and improve upon using more advanced machine learning tools.