Development of artificial neural network models for paediatric critical illness in South Africa

Abstract

Objectives: Failures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation.

Design: Prospective, analytical cohort study.

Setting: A single centre tertiary hospital in South Africa providing acute paediatric services.

Patients: Children, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations.

Outcomes: Predictive models for a composite outcome of death before discharge from hospital or admission to the PICU.

Interventions: None.

Measurements and main results: 765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit.

Conclusions: All models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.

Keywords: children; critical care; machine learning; neural networks; severity of illness; triage.

Figure 1

Consort diagram of development pipeline.

Figure 2

ANN model architectures – input layer = green, hidden layer = blue, output layer = red. A ReLu activation was used in the input and hidden layers and a sigmoid activation function in the output layer. ANN1 – Inclusive Model. ANN2 – Parsimonious Model, ANN3 – AIC Model. Figure compiled using ann_visualizer (51).

Figure 3

ROC (A) and PR (B) curves – AUC = area under the curve, CI = 95% confidence interval.

Figure 4

Normalised flexible calibration curve.

Figure 5

Decision Curve Analysis: (A) Net benefit compared to threshold probability; (B) Interventions avoided compared to threshold probability.