The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression

Abstract

Objective: Methods to correct class imbalance (imbalance between the frequency of outcome events and nonevents) are receiving increasing interest for developing prediction models. We examined the effect of imbalance correction on the performance of logistic regression models.

Material and methods: Prediction models were developed using standard and penalized (ridge) logistic regression under 4 methods to address class imbalance: no correction, random undersampling, random oversampling, and SMOTE. Model performance was evaluated in terms of discrimination, calibration, and classification. Using Monte Carlo simulations, we studied the impact of training set size, number of predictors, and the outcome event fraction. A case study on prediction modeling for ovarian cancer diagnosis is presented.

Results: The use of random undersampling, random oversampling, or SMOTE yielded poorly calibrated models: the probability to belong to the minority class was strongly overestimated. These methods did not result in higher areas under the ROC curve when compared with models developed without correction for class imbalance. Although imbalance correction improved the balance between sensitivity and specificity, similar results were obtained by shifting the probability threshold instead.

Discussion: Imbalance correction led to models with strong miscalibration without better ability to distinguish between patients with and without the outcome event. The inaccurate probability estimates reduce the clinical utility of the model, because decisions about treatment are ill-informed.

Conclusion: Outcome imbalance is not a problem in itself, imbalance correction may even worsen model performance.

Keywords: calibration; class imbalance; logistic regression; synthetic minority oversampling technique; undersampling.

Figure 1.

Visualization of how the imbalance correction methods work for a hypothetical dataset with 2 predictors. Black dots represent observations from the minority class, gray triangles represent observations from the majority class.

Figure 2.

Flexible calibration curves on the test set for the Ridge models to diagnose ovarian cancer.

Figure 3.

Decision curves on the test for the Ridge models to diagnose ovarian cancer. “All” refers to classifying all individuals as high risk (and hence treat all), “None” refers to classifying all individuals as low risk (and hence treat none).

Figure 4.

Test set AUROC for the Ridge models in the simulation scenarios with an event fraction of 1%.

Figure 5.

Test set calibration intercept for the Ridge models in the simulation scenarios with an event fraction of 1%.

Figure 6.

Test set calibration slope for the Ridge models in the simulation scenarios with an event fraction of 1%.