Deep-learning-based real-time prediction of acute kidney injury outperforms human predictive performance

Abstract

Acute kidney injury (AKI) is a major complication after cardiothoracic surgery. Early prediction of AKI could prompt preventive measures, but is challenging in the clinical routine. One important reason is that the amount of postoperative data is too massive and too high-dimensional to be effectively processed by the human operator. We therefore sought to develop a deep-learning-based algorithm that is able to predict postoperative AKI prior to the onset of symptoms and complications. Based on 96 routinely collected parameters we built a recurrent neural network (RNN) for real-time prediction of AKI after cardiothoracic surgery. From the data of 15,564 admissions we constructed a balanced training set (2224 admissions) for the development of the RNN. The model was then evaluated on an independent test set (350 admissions) and yielded an area under curve (AUC) (95% confidence interval) of 0.893 (0.862-0.924). We compared the performance of our model against that of experienced clinicians. The RNN significantly outperformed clinicians (AUC = 0.901 vs. 0.745, p < 0.001) and was overall well calibrated. This was not the case for the physicians, who systematically underestimated the risk (p < 0.001). In conclusion, the RNN was superior to physicians in the prediction of AKI after cardiothoracic surgery. It could potentially be integrated into hospitals' electronic health records for real-time patient monitoring and may help to detect early AKI and hence modify the treatment in perioperative care.

Keywords: Diagnosis; Preventive medicine.

Fig. 1. Experimental design for performance comparison of recurrent neural network (RNN) against physicians.

The electronic health record (EHR) data was split into a training and a test set. The training set was used for the development of the RNN (orange path). For each patient (Pat) in the test set, a quasi-random ‘prediction point’ in the time-series was chosen (for more information about quasi-randomness see ‘Methods’). EHR data up to this prediction point was given to physicians and RNN (the rest of the time series data, here denoted as X, was hidden). Both physicians and RNN, had to make a prediction for postoperative AKI at this prediction point.

Fig. 2. Discrimination and calibration of the predictions of recurrent neural network (RNN) and physicians.

a receiver operating characteristics (ROC), b precision-recall curve, c calibration of physicians, d calibration of RNN. AUC area under curve. H-L Hosmer-Lemeshow-Test, PR_AUC precision-recall AUC. The RNN outperformed clinical physicians regarding AUC (a) and PR_AUC (b). Physicians systematically underestimated the risk of acute kidney injury (predicted risks < observed risks, c). In contrast, the RNN was overall well calibrated (d).

Fig. 3. Flow chart of patient selection process.

adm admissions, pat patients.

Fig. 4. Total observation period for the training and test set.

a Density distribution. b Histogram. For most patients the observation period ended within three days after surgery.

Fig. 5. Architecture of a recurrent neural network (RNN).

At each time step, the model receives the current time slice data as input as well as the own output from the preceding time step. The features are captured in a truly sequential fashion.