Joint use of over- and under-sampling techniques and cross-validation for the development and assessment of prediction models

Abstract

Background: Prediction models are used in clinical research to develop rules that can be used to accurately predict the outcome of the patients based on some of their characteristics. They represent a valuable tool in the decision making process of clinicians and health policy makers, as they enable them to estimate the probability that patients have or will develop a disease, will respond to a treatment, or that their disease will recur. The interest devoted to prediction models in the biomedical community has been growing in the last few years. Often the data used to develop the prediction models are class-imbalanced as only few patients experience the event (and therefore belong to minority class).

Results: Prediction models developed using class-imbalanced data tend to achieve sub-optimal predictive accuracy in the minority class. This problem can be diminished by using sampling techniques aimed at balancing the class distribution. These techniques include under- and oversampling, where a fraction of the majority class samples are retained in the analysis or new samples from the minority class are generated. The correct assessment of how the prediction model is likely to perform on independent data is of crucial importance; in the absence of an independent data set, cross-validation is normally used. While the importance of correct cross-validation is well documented in the biomedical literature, the challenges posed by the joint use of sampling techniques and cross-validation have not been addressed.

Conclusions: We show that care must be taken to ensure that cross-validation is performed correctly on sampled data, and that the risk of overestimating the predictive accuracy is greater when oversampling techniques are used. Examples based on the re-analysis of real datasets and simulation studies are provided. We identify some results from the biomedical literature where the incorrect cross-validation was performed, where we expect that the performance of oversampling techniques was heavily overestimated.

Fig. 1

Combination of sampling and CV methods used in the simulations and real data analyses. CV includes Sampling (first row) constitutes the correct approach, while Sampling followed by CV (second row) is the incorrect approach. The samples included in the original dataset are indicated using upper cases, while their copies are indicated with lower cases

Fig. 2

Probability that at least one of the replicas of a sample included in the test fold is included also in the training fold, as a function of the proportion of minority class samples (p _min). The figure shows how the probability that a test sample has a replica in the learning fold depends on the level of class-imbalance (p _min) in a dataset with n=100 samples when using 2 fold CV

Fig. 3

Cross-validated AUC for different sample sizes and classification rules obtained on simulated data. AUC obtained with different classification rules for simulated data with 10 variables (simulated independently from a Gaussian distribution with zero mean and unit variance) and 2 CV folds. There were n=100, 500, 1,000 and 100,00 samples

Fig. 4

Cross-validated AUC for different UCI datasets. Datasets are ordered by their AUC obtained by correct CV

Fig. 5

Cross-validated AUC for different gene expression microarray datasets datasets. Datasets are ordered by their AUC obtained by correct CV