Avoiding common machine learning pitfalls

Abstract

Mistakes in machine learning practice are commonplace and can result in loss of confidence in the findings and products of machine learning. This tutorial outlines common mistakes that occur when using machine learning and what can be done to avoid them. While it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to make rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and how to report results.

Keywords: guidance; machine learning; practice.

Figure 1

See “do not allow test data to leak into the training process” (Left) How things should be, with the training set used to train the model and the test set used to measure its generality. (Right) When there is a data leak, the test set can implicitly become part of the training process, meaning that it no longer provides a reliable measure of generality.

Figure 2

See “do keep up with progress in deep learning (and its pitfalls)” A rough history of neural networks and deep learning showing what I consider to be the milestones in their development. For a far more thorough account of the field’s historical development, take a look at Schmidhuber.^,

Figure 3

See “do be careful where and how you do feature selection” (Top) Data leakage due to carrying out feature selection before splitting off the test data (outlined in red), causing the test set to become an implicit part of model training. (Middle) How it should be done. (Bottom) When using cross-validation, it is important to carry out feature selection independently for each iteration, based only on the subset of data (shown in blue) used for training during that iteration.

Figure 4

See “do avoid learning spurious correlations” The problem of spurious correlations in images as illustrated by the tank problem. The images on the left are tanks, and those on the right are not tanks. However, the consistent background (blue for tanks, gray for others) means that these images can be classified by merely looking at the colors of pixels toward the top of the images rather than having to recognize the objects in the images, resulting in a poor model.

Figure 5

See “do avoid sequential overfitting” (Top) Using the test set repeatedly during model selection results in the test set becoming an implicit part of the training process. (Bottom) A validation set should be used instead during model selection, and the test set should only be used once to measure the generality of the final model.

Figure 6

See “do choose metrics carefully” The problem with using accuracy as a performance metric on imbalanced data. Here, a dummy model that always predicts the same class label has an accuracy of 50% or 90% depending on the distribution of class labels within the data.

Figure 7

See “do not ignore temporal dependencies in time-series data” (Top) A time series is scaled to the interval $\left[0,1\right]$ before splitting off the test data (shown in red). This could allow the model to infer that values will increase in the future, causing a potential look ahead bias. (Bottom) Instead, the data should be split before doing scaling so that information about the range of the test data cannot leak into the training data.

Figure 8

See “do look at your models” Using saliency maps to analyze vision-based deep learning models. Imagine these two maps (in red) were generated for the image shown in the center, for two different deep learning models trained on the kind of tank recognition data mentioned in “do avoid learning spurious correlations.” Darker colors indicate features that are of greater importance to the model, so the model on the left (which predominantly focuses on the components of the tank) is likely to generalize much better than the one on the right (which predominantly focuses on the background of the image).