Classification of Current Procedural Terminology Codes from Electronic Health Record Data Using Machine Learning

Abstract

Background: Accurate anesthesiology procedure code data are essential to quality improvement, research, and reimbursement tasks within anesthesiology practices. Advanced data science techniques, including machine learning and natural language processing, offer opportunities to develop classification tools for Current Procedural Terminology codes across anesthesia procedures.

Methods: Models were created using a Train/Test dataset including 1,164,343 procedures from 16 academic and private hospitals. Five supervised machine learning models were created to classify anesthesiology Current Procedural Terminology codes, with accuracy defined as first choice classification matching the institutional-assigned code existing in the perioperative database. The two best performing models were further refined and tested on a Holdout dataset from a single institution distinct from Train/Test. A tunable confidence parameter was created to identify cases for which models were highly accurate, with the goal of at least 95% accuracy, above the reported 2018 Centers for Medicare and Medicaid Services (Baltimore, Maryland) fee-for-service accuracy. Actual submitted claim data from billing specialists were used as a reference standard.

Results: Support vector machine and neural network label-embedding attentive models were the best performing models, respectively, demonstrating overall accuracies of 87.9% and 84.2% (single best code), and 96.8% and 94.0% (within top three). Classification accuracy was 96.4% in 47.0% of cases using support vector machine and 94.4% in 62.2% of cases using label-embedding attentive model within the Train/Test dataset. In the Holdout dataset, respective classification accuracies were 93.1% in 58.0% of cases and 95.0% among 62.0%. The most important feature in model training was procedure text.

Conclusions: Through application of machine learning and natural language processing techniques, highly accurate real-time models were created for anesthesiology Current Procedural Terminology code classification. The increased processing speed and a priori targeted accuracy of this classification approach may provide performance optimization and cost reduction for quality improvement, research, and reimbursement tasks reliant on anesthesiology procedure codes.

Figure 1:. Machine learning study design flowchart.

Flow diagram of the experimental design of this study. The Train/Test data set is used to create each model while the Holdout data set is used as an external validation. Each model is trained using 5-fold cross validation. Parameter tuning occurred with each of the 20 iteration of model training. The single institution from the Holdout Dataset was not included in the 16 institutions included in the Train/Test Dataset.

Figure 2:. Accuracy of Current Procedural Terminology code assignment as a function of confidence parameter.

This graph shows the percentage accuracy of model CPT classification (y-axis) for a given cutoff of confidence parameter (x-axis) for the support vector machine model. The Train/Test and Holdout data set accuracies are plotted for both the first assigned CPT code (“top 1 CPT code”) and top three assigned CPT codes (“top 3 CPT codes”). High (≥1.6), Medium (1.6>confidence parameter≥1.2), Low (<1.2) areas are labeled above the figure. Confidence parameter is a derived measure of relative probability between best-fit and second best-fit CPT classifications. Current Procedural Terminology (CPT).

Figure 3:. Percentage case inclusion as a function of confidence parameter.

This graph shows the percentage of cases included for model CPT classification (y-axis) for a given cutoff of confidence parameter (x-axis) for the support vector machine model. The Train/Test and Holdout data set accuracies are plotted. High (≥1.6), Medium (1.6>confidence parameter≥1.2), Low (<1.2) areas are labeled above the figure. Confidence parameter is a derived measure of relative probability between best-fit and second best-fit CPT classifications. Current Procedural Terminology (CPT).