Jagged Cube Solved
The Jagged Cube Problem
This project ran into a major obstacle when trying to run K-Means Clustering or Principal Component Analysis on the NFL player career dataset. Both of these techniques require vectors to all be the same length, for proper comparison, and yet NFL careers are of considerably variable length. Many players play for just a few years and retire, while others play for over 20+ years! This poses a significant non-trivial issue in these two arenas. This issue has been playfully given the nickname "The Jagged Cube Problem", based on the fact that one dimension of the higher-dimensional data cube is of variable length. If it is helpful, this issue can be visualized in the imagination as follows:
For K-Means, the classical work-around was employed; that of 'padding' the shorter vectors with a dummy coefficient of some kind. -1 is very standard. While this is standard practice, it does tend to have the undesired consequence of skewing the clustering to make the most meaning part of the whole data interaction be those -1 padding elements. This is because the data has already been normalized to (in this case 'mostly') be in a range between 0 and 1. This makes the -1 elements, especially a large block of them, become the most dominant feature in the entire data set. The problem with this was quite visible by the end of the K-Means experiment, as the end result had considerably worse predictive capabilities than the simpler "collect the 30 closest players and take the median" method, the current high score:
Closest Player & Median
K-Means Clustering
For Principal Component Analysis, a new approach was pioneered of attempting to stretch out the NFL players's careers to all be of equal 100 year length. At best this would always have been a little shaky, as it's almost silly to compare a short 3 year career to a similarly-shaped 20 year career and say they're similar. But it caused dramatic new technical issues in the programming and made the issue of comparing a partially-complete player career to the known historical player career dataset nearly impossible. This issue was so dramatic, it ultimately led to no final PCA-version of the predictive engine even being completed at all. The complexity and runtime of the code simply exploded.
An innovative new solution was needed.
The Jagged Cube Problem Solved!
For the sake of simplicity, consider just K-Means for the moment. K-Means is a very effective algorithm for clustering data, in theory including NFL players (!), so it's a crying shame that so far it has not been successfully implemented into this project. K-Means algorithm is wonderful, reliable, and effective, and it would be a very good thing to have on one's side. It seemed to unlike the K-Means algorithm to have it, with the -1 padding of the vectors, produce such terrible results. It was very deeply desired to have some way for it to work with the Jagged Cube-shaped data.
After deep and careful consideration, a breakthrough occurred. The core issue that has been occurring up until now has been that it is impossible, as far as this author understands, to run K-Means on the entire dataset, since the data set has vectors of unequal length and thus the datacube has one jagged edge. But what if the project did the following instead:
Suppose the predictive engine begins with a given test player, T, and a given test number of years of career data, Z. The engine then finds the correct datacube for player T's position in the NFL, makes a temporary copy of that associated datacube, and performs the following transformations. It prunes from the data all players who played less than Z years. It then finds all the players who played for more than Z years and trims their careers down in length to just the first Z years. The temporary datacube thus formed is no longer jagged-edged! This means that K-Means (and also PCA!) could be performed upon the nice, pleasant, cube-shaped datacube which would no longer have the one dreaded jagged edge.
There is a significant downside of this operation, however, as once the new transformed no-longer-jagged-edge cube is formed, one cannot simply perform K-Means (or PCA) upon the entire cube, as the test player is currently included in that data. For scientific validity, one cannot make the clusters with the test player in the data, then remove them, and then try and find which cluser they best fit into. Their placement helped form that very cluster! This means that for each individual test player, the entire K-Means operation (and PCA process) must be re-done from scratch, very significantly boosting the duration of the runtime.
While not a perfect solution, this technique finally let's K-Means and Principal Component Analysis be performed upon the NFL dataset! More exciting still, it lets new versions of the predictive engine be designed to use those two techniques as part of their method of operation.
K-Means Clustering
Using the above new idea, it was finally possibe to run a full, deep, rich K-Means Clustering-based version of the NFL predictive engine! A number of smaller Silhouette tests were performed upon the new, transformed, non-jagged datacubes that indicated that the best choice for K was somewhere around K=20 for each of the datasets. For the sake of simplicity, it was decided to run all the datasets with K=20, rather than having slightly different K-values for each player position and each number of years.
The final result of the experiment was the following:
K-Means Clustering
This is a vast improvement on the previous version of K-Means that used the -1 padding! The mean retirement error improved by 1.4163 years, while the mean prediction error dropped by only 0.0033 years. This is a remarkable change in a positive direction.
While it was originally incredibly exciting that this new solution for the jagged cube problem might offer a way to beat the high score, the original 30-closest-neighbors method (armed with median) still beats this retirement error by an average of 0.1482 years and this prediction error by an average of 0.025 percentile points.
None the less, it was still incredibly catharitic to see K-Means finally show its prowess and present a quite-reasonable way of constructing an NFL player career predictive machine.
Principal Component Analysis
Just like with K-Means, the solving of The Jagged Cube Problem meant that the project could finally also move forward with PCA as a technique to incorporate into the predictive engine (as well as possibly any future machine learning methods that require vectors of all equal length). Since K-Means proved to be mildly less effective at predicting than using the original 30-closest-neighbors method, with median, the PCA version of the engine was designed to work as follows:
For each test player and given number of years of career data, the datacube would once again be transformed to be of all equal career lengths, thus resulting in a perfectly cube-shaped cube with no more jagged edge. At this point, PCA analysis would be performed upon that newly transformed cube. The PCA technique, in the best of world, might let the predictive engine tune out some of the 'noise' in the data, and perhaps let it see more clearly. At that point, the engine would once again find the 30 closest players and produce its predictions based upon the median of those players's careers.
The core remaining question left was what percent of the variance should the PCA technique be designed to keep? 95% is often the standard, but it was worth testing a variety of values to see which would produce the best predictions. A special Python code was written (with a new random seed) to test just this and resulted in the following data:
For a moment, this was incredibly promising! While the mean prediction error did not beat the current best known technique, the mean retirement error of 1.69 years was quite a lot better than the current high score of 1.83! The full PCA version of the predictive engine was designed, coded, and ran in such a way as to keep 85% of the original variance in an attempt to perhaps generate, specifically, a highly effective prediction for retirement. However, when the results were finally in, tested according to the baseline test of 10000 trials with random.seed(333), it proved to not be as exciting as originally hoped.
PCA
While ultimately not an improvement on the highscore, the earlier experiment leaves room for the fact that it is possible that with careful use of PCA techniques, it might be possible to improve the predictive engine and beat placebo.
Conclusions
The primary accomplishment of this work was not that K-Means or Principal Component Analysis ultimately outperformed the existing preditive engine, but that it became possible to apply those techniques to the NFL career dataset at all. By transforming each datacube temporarily into a new trimmed datacube wherein all players have identical observed career lengths, the longstanding "Jagged Cube Problem" was eliminated without relying on artificial padding values that distorted the geometry of the data.
In a sense, solve the Jagged Cube Problem may ultimately prove to be the more valuable contribution that any single predictive result, as it dramatically expands the range of machine learning techniques that can now be explored on variable-length career data in future work.