Validating the early phototherapy prediction tool across cohorts

Abstract

Background: Hyperbilirubinemia of the newborn infant is a common disease worldwide. However, recognized early and treated appropriately, it typically remains innocuous. We recently developed an early phototherapy prediction tool (EPPT) by means of machine learning (ML) utilizing just one bilirubin measurement and few clinical variables. The aim of this study is to test applicability and performance of the EPPT on a new patient cohort from a different population.

Materials and methods: This work is a retrospective study of prospectively recorded neonatal data from infants born in 2018 in an academic hospital, Regensburg, Germany, meeting the following inclusion criteria: born with 34 completed weeks of gestation or more, at least two total serum bilirubin (TSB) measurement prior to phototherapy. First, the original EPPT-an ensemble of a logistic regression and a random forest-was used in its freely accessible version and evaluated in terms of the area under the receiver operating characteristic curve (AUROC). Second, a new version of the EPPT model was re-trained on the data from the new cohort. Third, the predictive performance, variable importance, sensitivity and specificity were analyzed and compared across the original and re-trained models.

Results: In total, 1,109 neonates were included with a median (IQR) gestational age of 38.4 (36.6-39.9) and a total of 3,940 bilirubin measurements prior to any phototherapy treatment, which was required in 154 neonates (13.9%). For the phototherapy treatment prediction, the original EPPT achieved a predictive performance of 84.6% AUROC on the new cohort. After re-training the model on a subset of the new dataset, 88.8% AUROC was achieved as evaluated by cross validation. The same five variables as for the original model were found to be most important for the prediction on the new cohort, namely gestational age at birth, birth weight, bilirubin to weight ratio, hours since birth, bilirubin value.

Discussion: The individual risk for treatment requirement in neonatal hyperbilirubinemia is robustly predictable in different patient cohorts with a previously developed ML tool (EPPT) demanding just one TSB value and only four clinical parameters. Further prospective validation studies are needed to develop an effective and safe clinical decision support system.

Keywords: baby; children; jaundice; machine learning; prediction.

Figure 1

Evaluation of the predictive performance of the EPPT on the KUNO dataset. We compare the original EPPT trained on the UKBB dataset with the same model trained on the KUNO dataset. The bold lines show the operator characteristic (ROC) curve for the respective models. The fine lines show the ROC curve for the individual cross-validation folds of the re-trained model.

Figure 2

Variable importance values for the random forest trained using all variables in the dataset. Each bar denotes the relative importance of the respective variable and the standard deviation across cross validation folds is shown with error bars. The results indicate that the predictive performance of the random forest depends mostly on a subset of five variables, which correspond to the same variables used by the original EPPT.

Figure 3

Backwards feature selection using the random forest. Starting with all variables, in each step we remove the variable with lowest variable importance. Markers denote the average AUROC across 20 cross-validation folds and error bars denote the standard deviation respectively. The last six variables removed were the following (in the given order): hours since previous measurement, bilirubin, hours since birth, gestational age, birth weight, bilirubin to weight ratio.

Figure 4

Sensitivity, specificity, and positive predictive value (PPV) for the re-trained EPPT as a function of the decision threshold value. Error bars denote the standard deviation across 20 cross-validation folds. PPV values for threshold values larger than 0.575 are missing, because there were individual folds without positive predictions as the predicted probabilities did not exceed the threshold value.