Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Nov 25;40(3):e1398.
doi: 10.7196/SAJCC.2024.v40i3.1398. eCollection 2024.

A validation of machine learning models for the identification of critically ill children presenting to the paediatric emergency room of a tertiary hospital in South Africa: A proof of concept

M A Pienaar  1 , N Luwes  2 , J B Sempa  3 , E George  4 , S C Brown  5
Affiliations

A validation of machine learning models for the identification of critically ill children presenting to the paediatric emergency room of a tertiary hospital in South Africa: A proof of concept

M A Pienaar et al. South Afr J Crit Care. .

Abstract

Background: Machine learning (ML) refers to computational algorithms designed to learn from patterns in data to provide insights or predictions related to that data.

Objectives: Multiple studies report the development of predictive models for triage or identification of critically ill children. In this study, we validate machine learning models developed in South Africa for the identification of critically ill children presenting to a tertiary hospital.

Results: The validation sample comprised 267 patients. The event rate for the study outcome was 0.12. All models demonstrated good discrimination but weak calibration. Artificial neural network 1 (ANN1) had the highest area under the receiver operating characteristic curve (AUROC) with a value of 0.84. ANN2 had the highest area under the precision-recall curve (AUPRC) with a value of 0.65. Decision curve analysis demonstrated that all models were superior to standard strategies of treating all patients or treating no patients at a proposed threshold probability of 10%. Confidence intervals for model performance overlapped considerably. Post hoc model explanations demonstrated that models were logically coherent with clinical knowledge.

Conclusions: Internal validation of the predictive models correlated with model performance in the development study. The models were able to discriminate between critically ill children and non-critically ill children; however, the superiority of one model over the others could not be demonstrated in this study. Therefore, models such as these still require further refinement and external validation before implementation in clinical practice. Indeed, successful implementation of machine learning in practice within the South African setting will require the development of regulatory and infrastructural frameworks in conjunction with the adoption of alternative approaches to electronic data capture, such as the use of mobile devices.

Keywords: Machine learning; children; critical care; domain knowledge; severity of illness (min. 5 - max. 8); triage.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Receiver operating characteristic (A) and precision recall (B) curves.
Fig. 2
Fig. 2
Decision curve analysis
Fig. 3
Fig. 3
SHAP feature importances. (SPO = peripheral pulse oximetry measurement.)
See this image and copyright information in PMC

References

    1. Turner EL, Nielsen KR, Jamal SM, von Saint A, Musa NL. A review of pediatric critical care in resource-limited settings: A look at past, present, and future directions. Front Pediatr. 2016;4(Feb):1–15. - PMC - PubMed
    1. Hodkinson P, Argent A, Wallis L, et al. Pathways to care for critically ill or injured children: A cohort study from first presentation to healthcare services through to admission to intensive care or death. PLoS One. 2016;11(1):1–17. - PMC - PubMed
    1. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. 2019;380(14):1347–1358. - PubMed
    1. Goto T, Camargo Jr CA, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning-based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. 2019;4 - PMC - PubMed
    1. Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019;23(1):1–13. - PMC - PubMed

LinkOut - more resources