A machine learning approach to triaging patients with chronic obstructive pulmonary disease

Abstract

COPD patients are burdened with a daily risk of acute exacerbation and loss of control, which could be mitigated by effective, on-demand decision support tools. In this study, we present a machine learning-based strategy for early detection of exacerbations and subsequent triage. Our application uses physician opinion in a statistically and clinically comprehensive set of patient cases to train a supervised prediction algorithm. The accuracy of the model is assessed against a panel of physicians each triaging identical cases in a representative patient validation set. Our results show that algorithm accuracy and safety indicators surpass all individual pulmonologists in both identifying exacerbations and predicting the consensus triage in a 101 case validation set. The algorithm is also the top performer in sensitivity, specificity, and ppv when predicting a patient's need for emergency care.

Fig 1. Process for generating patient case scenarios and collecting algorithm training and validation data.

Fig 2. Training, validation, and optimization procedure for building COPD exacerbation and triage prediction algorithms.

Fig 3. Comparison between ML classifiers at matching consensus decision in the validation set.

SVMP, SVML, and SVMG are all support vector machine algorithms with polynomial, linear, and Gaussian kernels respectively. RF = Random Forest, NB = Naïve Bayes, LR = Logistic Regression, KNN = K-Nearest Neighbors, GB = Gradient Boosted Random Forest, and ET = Extra Decision Tree Classifier.

Fig 4. Performance comparison when the algorithm and all of the physicians got a vote in the consensus opinion.

Comparison of the algorithm and individual physicians at predicting the consensus triage and exacerbation (y/n) in the validation set: (a) triage identification, (b) exacerbation identification. A comparison of the algorithm with the average physician in accuracy, sensitivity, specificity, ppv, and npv for: (c) triage identification, (d) exacerbation identification. Triage statistics were computed as defined in Eqs 1–7.

Fig 5. Performance comparison of algorithm and individual physicians at predicting the consensus of the validation sets.

(a) triage performance, algorithm was not included in consensus, (b) exacerbation performance, algorithm was not included in consensus. (c) triage performance, no member votes when assessing their accuracy, (d) exacerbation performance, no member votes when assessing their accuracy.

Fig 6. Confusion matrices comparing assessment performance of the GB algorithm to the top physician.

(a) triage, (b) exacerbation. Note: top physician = the physician with highest classification accuracy.

Fig 7. Comparing the performance of the GB algorithm to the top physician in assessing the need for medical attention.

Note: top physician = physician with highest classification accuracy.

Fig 8. Distributions for each physician in the validation set (left) and the averaged distributions (right).

(a) triage distribution, (b) averaged triage distribution, (c) exacerbation distribution, (d) averaged exacerbation distribution. Note: error bars indicate 1 standard deviation about the mean.

Fig 9. Plot of % change in consensus triage answers as additional doctors are added to the validation panel (plus algo).

The average change when the panel reaches 10 members (from 9) is 5.5%.