Neural Networks
Overview
A neural network (NN) is a supervised machine learning model that learns complex relationships between input variables and desired outputs through numerical optimization. Rather than defining explicit decision rules, the model learns a set of weights from labeled training data that minimize prediction error. During training, the network repeatedly processes examples from the training set, computes prediction errors, and adjusts its weights using backpropagation and gradient descent. This process is repeated over multiple epoch cycles until the model converges to a satisfactory solution.
Neural networks are particularly effective for problems in which the relationship between the input variables and the desired output is nonlinear or difficult to model analytically. By combining multiple layers of weighted computations with nonlinear activation functions, the network can learn decision boundaries and predictive relationships that may not be captured by simpler machine learning algorithms.
The basic operations of a neural network looks a little like something like this:
Information enters through the input layer, where each node represents one feature of the input data. The values are then passed through one or more hidden layers, where each connection has an associated numerical weight. At every hidden node, the weighted inputs are combined and transformed using a specific, pre-selected activation function before being passed to the next layer. During training, the network compares its predictions to the known labels, calculates the resulting error, and uses backpropagation to calculate that error backward through the network. This process updates the connection weights using gradient descent so that future predictions become increasingly accurate over multiple training cycles. Finally, the processed information reaches the output layer, where the network produces its prediction.
When working with neural networks, these are the three most commonly selected activation functions:
For this project, a neural network will be trained to model the progression of NFL player careers using historical percentile scores from previous seasons. The core of the approach will mimic the recursive style of classification that performed so well for the SVM version of the machine. Given a player's career up through a given number of years, Z, the network will first classify whether the player is expected to continue playing or retire before season Z+1. If the player is predicted to continue, the network will then predict the player's performance for season Z+1. This predicted percentile is appended to the player's career history and fed back into the model, allowing the prediction process to continue recursively for seasons Z+2, Z+3, etc. The process terminates only when the neural network predicts that the player has retired, thereby generating an estimated remainder of the player's career.
Data Prep
Since neural networks are a supervised machine learning technique, this means that they require a collection of labeled training examples from which to learn. In this project, for each player the input feature vector will be a career-simplified percentiles only vector (like the one used for SVM), and the output (aka label) will be either whether or not that player retired the next year or what bin to assign their next season's performance percentile into. During the neural network training, the NN repeatedly compares its predictions to known labels, computes the resulting prediction error, and adjusts its internal weights through backpropagation and gradient descent. Because the correct outputs are known during training, the network is able to gradually learn relationships between the input data and the desired classifications.
During the SVM portion of the project, the full player career datacubes were trimmed down all the way to their absolute bare bones to show only the career percentiles for each year that each player played. Here's a glimpse inside the trimmed QB datacube:
To train the neural network, new datacubes needed to be created. For each player position there would be a unique data cube for each number of valid years (3+) of input data. Each row corresponds to a single player, each column shows how they played in years 1, 2, 3, etc of their career. And finally all the way on the right, the last two columns show the labels: did they play the next season (recorded as a 0) or retire (recorded as a 1) and if they did play, what was their performance percentile for that next season?
Here is a glimpse inside that new training datacube for the pair (QB, 5 years):
Once the labeled datasets were created, each was randomly divided into two disjoint subsets. 80% of each dataset was assigned to the training set, 20% was assigned to the testing set. The training set exclusively would be used to train the NN. After the training was complete, the test set would be presented to the NN for the first time to evaluate its predictive performance on previously unseen data.
It is imperative that the two datasets remain disjoint. If not, then some players would appear in both sets, invalidating the research. It would mean that some of the test cases would be literally the same material the NN was directly trained on and would artificially inflate the accuracy score. By evaluating the model only on players that were withheld from training, the resulting accuracy provides a more realistic measure of how well the neural network is expected to generalize to new NFL players whose careers were not used during the learning process.
Before
After
Code
At this stage, a first draft of the neural network was designed and implemented. It was made using the Tensorflow library. This particular NN implementation would test out how to get a neural network to do just the first year's retirement prediction. All the rest would be simple to program using this example and a looped code structure.
A simple feedforward NN was selected for simplicity. The nework architecture consisted of a single hidden layer containing 16 hidden units armed with the ReLU activation function, followed by a single output neuron armed with the Sigmoid activation function. The network was trained using binary cross entropy loss over 50 training epochs.
The first draft was constructed and tested out ... but the results were disastrous. It never predicted a single player to retire, ever. Not before their 20th season at least. Because players much more frequently continue their careers than do they retire, the neural network learned to classify nearly every player as returning their following season. But then it was recalled that this same exact design flaw had been encountered when programming the SVM classification engine. With SVM, it was possible to assign a weight value to one or more bins, making the classifications lean in that direction. Some research later, it was found that a similar weight value could be applied towards retirement here with neural networks.
A retirement weighting parameter was introduced during training. Incorrect retirement classifications were assigned progressively larger penalties, encouraging the network to learn a more meaningful decision boundary. Ten separate experiments were performed using retirement weights ranging from 1 through 10, while keeping every other network parameter fixed, including a set random seed to ensure proper scientific legitimacy. The resulting classification accuracy and confusion matrices were recorded for each experiment. The results were as follows:
As seen in the above graphs, too-small a retirement weight caused the network to classify nearly every player as continuing to play, producing a somewhat artificial high accuracy but failing to identify retirement players entirely. Conversly, too-large a retirement weight casted the network to overpredict retirements, substantially increasing the number of false retirement predictions. A retirement weight of 4 proved to be the best middle ground, providing the highest classification accuracy and simultaneously identifying several retiring players with relatively few false retirement predictions. This value was thus chosen to be the retirement weight for all remaining neural network experiments going forward in this project.
A larger-scale experiment was run, to fully test out the neural networks that had been trained. 10000 random player/year pairs were selected (with replacement) from the test group to see how well the NN would perform. For each randomly selected player, the appropriate neural network—based on player position and number of revealed career seasons—was used to predict whether the player would retire before the following season. These predictions were then compared to the known outcomes recorded in the testing data. Overall classification accuracy was computed, and the resulting predictions were summarized using a confusion matrix. The final five training epochs of the network are also included to illustrate the convergence of the training process. The resulting outputs are shown below.
These results were, to say the least, quite disappointing. The accuracy was profoundly off. Overall classification accuracy was substantially lower than anticipated, indicating that the neural network was not performing nearly as well as the previously developed Support Vector Machine models. Rather than immediately abandoning the approach, additional analysis was conducted to determine the source of the poor performance. Individual neural networks were retrained and evaluated separately for every player position and every career length represented in the dataset. This diagnostic process revealed that the neural networks performed quite well for quarterbacks and kickers, but struggled considerably when modeling rushing, receiving, and defensive players, particularly during the later stages of their careers. The resulting position-by-position accuracies are shown below.
Further analysis revealed that several factors contributed to the disappointing overall performance of the neural networks. First, the retirement weight of 4 had originally been selected by experimenting with quarterback data and was found to work well for that position. Unlike the Support Vector Machine implementation, however, this parameter did not generalize effectively across all player positions and career stages. Optimizing the neural networks would likely require selecting a separate retirement weight for every combination of player position and revealed career length. While such tuning could potentially improve performance, it would require dozens of independently optimized parameters and would begin to risk overfitting the available data. Second, a more fundamental limitation arose from the size of the training datasets themselves. As careers become longer, particularly for positions such as running back and defensive player where retirements occur earlier on average, the number of remaining players decreases dramatically. Some of the later-career datasets contained only a handful of training examples. Neural networks generally require substantially larger training sets to learn reliable decision boundaries, and their performance deteriorated considerably on these very small datasets.
Given the limited time remaining for the project, it was decided to incorporate neural networks only for the player positions where they demonstrated consistently strong performance: quarterbacks and kickers. These two positions achieved substantially higher testing accuracies than the remaining positions and therefore would provide a meaningful improvement to the overall predictive system. The above experiment was rerun on just quarterbacks and kickers with the following results:
These results are far more up to the expectations of the quality of this project so far and showed promise that a neural network-based version of the NFL predictive machine might be possible.
Results
The final version of the NN-based predictive engine (again, only for quarterbacks and kickers) was finally ready to be tested out. This hybrid architecture was chosen to allow incremental integration of neural networks while maintaining compatibility with the broader prediction pipeline. The predictive engine was finally ready for its ultimate evaluation. The results were as follows:
These results, while honest, were not very inspiring. Considering that the only changes were to quarterbacks and to kickers, the fairly dramatic worsening of the results implied that the negative change in results only for those players using the NN system must have been quite dramatic indeed.
Rather than producing an incremental improvement over the original approach, as was hoped, the neural network models appeared to significantly reduce the quality of both retirement prediction and the future-career percentile estimations as well. This suggests that, at least in their current form, the neural networks were not able to capture the underlying relationships in the data as effectively as the established 30-nearest-neighbor approach.
Several factors may have contributed to this outcome. Unlike the 30-nearest-neighbor method, which directly leverages historical player similarity, the neural networks relied on relatively limited training datasets for each position and career length. Splitting the data into many separate models reduced the amount of training data available to each individual network, making it more difficult for the models to generalize effectively. Neural Networks are not designed to operate on such tiny data sets as just those quarterbacks who played for, for instance, more than 15 years. It is highly probable that the NN results were quite skewed by certain extreme results (or possibly even attractive cycles) that pulled many of the predictions quite far off track.
Future work could explore larger networks, additional input features, more extensive hyperparameter tuning, or combining neural-network predictions with the nearest-neighbor methodology to determine whether a hybrid approach can achieve better predictive performance.
Conclusions
Overall, the neural network portion of this project demonstrated both the exciting potential and the challenges of replacing more traditional machine learning methods with a more cutting edge model. The highly simplistic 30-nearest neighbors model requires very little specialized training to run successfully, yet performs almost as well as the advanced SVM recursive model. In trying to integrate neural networks, there was significant dissonance between how NNs traditionally are used and how this specific project (predicting the future of multi-length vectors) needs to operate. The project was forced to back the NNs into tighter and tighter corners — smaller and smaller datasets — until they simply did not have enough training data to operate upon. This outcome is itself a meaningful result, illustrating that more sophisticated models do not necessarily produce better predictions when training data are limited or when the problem structure is better suited to similarity-based approaches.
Despite the disappointing predictive performance, this phase of the project provided valuable insight into the strengths and limitations of neural networks in the context of long-term NFL career forecasting. More broadly, this project demonstrated the importance of objectively evaluating new machine learning techniques rather than assuming they will outperform simpler alternatives. In this case, the 30-nearest-neighbor approach remained the stronger predictive model, reinforcing the principle that the most effective solution is determined by empirical performance rather than algorithmic complexity.