Natural language processing augments comorbidity documentation in neurosurgical inpatient admissions

Abstract

Objective: To establish whether or not a natural language processing technique could identify two common inpatient neurosurgical comorbidities using only text reports of inpatient head imaging.

Materials and methods: A training and testing dataset of reports of 979 CT or MRI scans of the brain for patients admitted to the neurosurgery service of a single hospital in June 2021 or to the Emergency Department between July 1-8, 2021, was identified. A variety of machine learning and deep learning algorithms utilizing natural language processing were trained on the training set (84% of the total cohort) and tested on the remaining images. A subset comparison cohort (n = 76) was then assessed to compare output of the best algorithm against real-life inpatient documentation.

Results: For "brain compression", a random forest classifier outperformed other candidate algorithms with an accuracy of 0.81 and area under the curve of 0.90 in the testing dataset. For "brain edema", a random forest classifier again outperformed other candidate algorithms with an accuracy of 0.92 and AUC of 0.94 in the testing dataset. In the provider comparison dataset, for "brain compression," the random forest algorithm demonstrated better accuracy (0.76 vs 0.70) and sensitivity (0.73 vs 0.43) than provider documentation. For "brain edema," the algorithm again demonstrated better accuracy (0.92 vs 0.84) and AUC (0.45 vs 0.09) than provider documentation.

Discussion: A natural language processing-based machine learning algorithm can reliably and reproducibly identify selected common neurosurgical comorbidities from radiology reports.

Conclusion: This result may justify the use of machine learning-based decision support to augment provider documentation.

Fig 1. Example of frequency and TF-IDF tokenization strategies illustrating how TF-IDF controls for words that frequently occur in the corpus.

TF-IDF = Term Frequency Inverse Document Frequency.

Fig 2. Machine learning and deep learning model performance on brain compression data.

(A) Machine learning classifiers’ performance methods with both frequency (TF) and term frequency-inverse document frequency (TFIDF) tokenization strategies. (B) Deep learning classifiers’ performance methods with both frequency and term frequency-inverse document frequency (TFIDF) tokenization strategies. SVM = support vector machine; NB = Naïve bayes; Log = Logistic regression.

Fig 3. Machine learning and deep learning model performance on brain edema data.

(A) Machine learning classifiers’ performance methods with both frequency (TF) and term frequency-inverse document frequency (tfidf) tokenization strategies. (B) Deep learning classifiers’ performance methods with both frequency and term frequency-inverse document frequency (TFIDF) tokenization strategies. SVM = support vector machine; NB = Naïve bayes; Log = Logistic regression.

Fig 4. Receiver operating characteristic (ROC) curves for random forest classifier with TF-IDF tokenization.

(A) Estimator trained for brain compression classification. (B) Estimator trained for brain edema classification. AUC = area under the curve.

Fig 5. Machine learning estimator and provider documentation comparison.

(A) Estimators for compression dataset. (B) Estimators for edema dataset. SVM = support vector machine; NB = Naïve bayes; Log = Logistic regression.