Machine learning-enabled maternal risk assessment for women with pre-eclampsia (the PIERS-ML model): a modelling study

Abstract

Background: Affecting 2-4% of pregnancies, pre-eclampsia is a leading cause of maternal death and morbidity worldwide. Using routinely available data, we aimed to develop and validate a novel machine learning-based and clinical setting-responsive time-of-disease model to rule out and rule in adverse maternal outcomes in women presenting with pre-eclampsia.

Methods: We used health system, demographic, and clinical data from the day of first assessment with pre-eclampsia to predict a Delphi-derived composite outcome of maternal mortality or severe morbidity within 2 days. Machine learning methods, multiple imputation, and ten-fold cross-validation were used to fit models on a development dataset (75% of combined published data of 8843 patients from 11 low-income, middle-income, and high-income countries). Validation was undertaken on the unseen 25%, and an additional external validation was performed in 2901 inpatient women admitted with pre-eclampsia to two hospitals in south-east England. Predictive risk accuracy was determined by area-under-the-receiver-operator characteristic (AUROC), and risk categories were data-driven and defined by negative (-LR) and positive (+LR) likelihood ratios.

Findings: Of 8843 participants, 590 (6·7%) developed the composite adverse maternal outcome within 2 days, 813 (9·2%) within 7 days, and 1083 (12·2%) at any time. An 18-variable random forest-based prediction model, PIERS-ML, was accurate (AUROC 0·80 [95% CI 0·76-0·84] vs the currently used logistic regression model, fullPIERS: AUROC 0·68 [0·63-0·74]) and categorised women into very low risk (-LR <0·1; eight [0·7%] of 1103 women), low risk (-LR 0·1 to 0·2; 321 [29·1%] women), moderate risk (-LR >0·2 and +LR <5·0; 676 [61·3%] women), high risk (+LR 5·0 to 10·0, 87 [7·9%] women), and very high risk (+LR >10·0; 11 [1·0%] women). Adverse maternal event rates were 0% for very low risk, 2% for low risk, 5% for moderate risk, 26% for high risk, and 91% for very high risk within 48 h. The 2901 women in the external validation dataset were accurately classified as being at very low risk (0% with outcomes), low risk (1%), moderate risk (4%), high risk (33%), or very high risk (67%).

Interpretation: The PIERS-ML model improves identification of women with pre-eclampsia who are at lowest and greatest risk of severe adverse maternal outcomes within 2 days of assessment, and can support provision of accurate guidance to women, their families, and their maternity care providers.

Funding: University of Strathclyde Diversity in Data Linkage Centre for Doctoral Training, the Fetal Medicine Foundation, The Canadian Institutes of Health Research, and the Bill & Melinda Gates Foundation.

Figure 1

Shapley values for the PIERS-ML variables in a single random forest on the corresponding development dataset Shapley values represent the feature value's contribution to the individual's predicted probability. Positive Shapley values increase the predicted probability, negative values decrease the predicted probability, and values of zero show no change to the predicted probability. EDD=expected date of delivery. GDP=gross domestic product. SpO₂=oxygen saturation by pulse oximetry.

Figure 2

PIERS-ML variables ranked by importance within the random forest model based on Gini index, compared with the least important variable (National MMR) Random Forests enable examination of feature importances, which is the mean of the amount the Gini Index (or node impurity) decreases by in each tree at the split that uses the feature. The more the Gini Index decreases for a feature, the more important it is. This figure rates the features from 0–100, with 100 being the most important. EDD=expected date of delivery. GDP=gross domestic product. SpO₂=oxygen saturation by pulse oximetry.

Figure 3

Calibration plot for the PIERS-ML model Predictions on the validation and testing datasets, in increasing order, were binned together into ten groups of 221 predictions. Event rates (observed risk) were calculated along with confidence intervals and plotted against the mean predicted probability per group to create the dot and whisker plot. Smooth lines were plotted using the individual predicted probabilities and yes and no outcomes, with Linear (red) and Loess (blue) methods. Cox calibration intercept and slope, Brier score, and Spiegelhalter Z scores were calculated.

Figure 4

PIERS-ML area under receiver-operator characteristic for adverse maternal outcomes within 2 days of initial assessment using data within 1 day of initial assessment and before the occurrence of any outcome Area-under-the receiver-operator characteristic of 0·78 (95% CI 0·73–0·82).

Figure 5

Precision-recall plot for the PIERS-ML model