Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 May 31;6(2):ooad033.
doi: 10.1093/jamiaopen/ooad033. eCollection 2023 Jul.

Implications of resampling data to address the class imbalance problem (IRCIP): an evaluation of impact on performance between classification algorithms in medical data

Koen Welvaars  1 , Jacobien H F Oosterhoff  2 , Michel P J van den Bekerom  3   4 , Job N Doornberg  5 , Ernst P van Haarst  6 ; OLVG Urology Consortium, and the Machine Learning Consortium
Collaborators, Affiliations

Implications of resampling data to address the class imbalance problem (IRCIP): an evaluation of impact on performance between classification algorithms in medical data

Koen Welvaars et al. JAMIA Open. .

Abstract

Objective: When correcting for the "class imbalance" problem in medical data, the effects of resampling applied on classifier algorithms remain unclear. We examined the effect on performance over several combinations of classifiers and resampling ratios.

Materials and methods: Multiple classification algorithms were trained on 7 resampled datasets: no correction, random undersampling, 4 ratios of Synthetic Minority Oversampling Technique (SMOTE), and random oversampling with the Adaptive Synthetic algorithm (ADASYN). Performance was evaluated in Area Under the Curve (AUC), precision, recall, Brier score, and calibration metrics. A case study on prediction modeling for 30-day unplanned readmissions in previously admitted Urology patients was presented.

Results: For most algorithms, using resampled data showed a significant increase in AUC and precision, ranging from 0.74 (CI: 0.69-0.79) to 0.93 (CI: 0.92-0.94), and 0.35 (CI: 0.12-0.58) to 0.86 (CI: 0.81-0.92) respectively. All classification algorithms showed significant increases in recall, and significant decreases in Brier score with distorted calibration overestimating positives.

Discussion: Imbalance correction resulted in an overall improved performance, yet poorly calibrated models. There can still be clinical utility due to a strong discriminating performance, specifically when predicting only low and high risk cases is clinically more relevant.

Conclusion: Resampling data resulted in increased performances in classification algorithms, yet produced an overestimation of positive predictions. Based on the findings from our case study, a thoughtful predefinition of the clinical prediction task may guide the use of resampling techniques in future studies aiming to improve clinical decision support tools.

Keywords: ADASYN; RUS; SMOTE; class imbalance; classification algorithms; resampling.

PubMed Disclaimer

Conflict of interest statement

All authors have no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc.) that might pose a conflict of interest in connection with the submitted article.

Figures

Figure 1.
Figure 1.
Process of under- and oversampling.
Figure 2.
Figure 2.
Flowchart of resampling strategy.
Figure 3.
Figure 3.
Evaluation metrics per learner.
Figure 4.
Figure 4.
Calibration curve plots per algorithm per sampled dataset.
See this image and copyright information in PMC

References

    1. Megahed FM, Chen Y-J, Megahed A, et al.The class imbalance problem. Nat Methods 2021; 18 (11): 1270–2. - PubMed
    1. Fernández A, García S, Galar M, et al.Learning from Imbalanced Data Sets. Cham: Springer International Publishing; 2018. doi: 10.1007/978-3-319-98074-4. - DOI
    1. Kim M, Hwang K-B.. An empirical evaluation of sampling methods for the classification of imbalanced data. PLoS One 2022; 17 (7): e0271260. - PMC - PubMed
    1. Li D-C, Liu C-W, Hu SC.. A learning method for the class imbalance problem with medical data sets. Comput Biol Med 2010; 40 (5): 509–18. - PubMed
    1. Fujiwara K, Huang Y, Hori K, et al.Over- and under-sampling approach for extremely imbalanced and small minority data problem in health record analysis. Front Public Health 2020; 8: 178. - PMC - PubMed

LinkOut - more resources