Machine learning optimization of an electronic health record audit for heart failure in primary care

Abstract

Aims: The diagnosis of heart failure (HF) is an important problem in primary care. We previously demonstrated a 74% increase in registered HF diagnoses in primary care electronic health records (EHRs) following an extended audit procedure. What remains unclear is the accuracy of registered HF pre-audit and which EHR variables are most important in the extended audit strategy. This study aims to describe the diagnostic HF classification sequence at different stages, assess general practitioner (GP) HF misclassification, and test the predictive performance of an optimized audit.

Methods and results: This is a secondary analysis of the OSCAR-HF study, a prospective observational trial including 51 participating GPs. OSCAR used an extended audit based on typical HF risk factors, signs, symptoms, and medications in GPs' EHR. This resulted in a list of possible HF patients, which participating GPs had to classify as HF or non-HF. We compared registered HF diagnoses before and after GPs' assessment. For our analysis of audit performance, we used GPs' assessment of HF as primary outcome and audit queries as dichotomous predictor variables for a gradient boosted machine (GBM) decision tree algorithm and logistic regression model. Of the 18 011 patients eligible for the audit intervention, 4678 (26.0%) were identified as possible HF patients and submitted for GPs' assessment in the audit stage. There were 310 patients with registered HF before GP assessment, of whom 146 (47.1%) were judged not to have HF by their GP (over-registration). There were 538 patients with registered HF after GP assessment, of whom 374 (69.5%) did not have registered HF before GP assessment (under-registration). The GBM and logistic regression model had a comparable predictive performance (area under the curve of 0.70 [95% confidence interval 0.65-0.77] and 0.69 [95% confidence interval 0.64-0.75], respectively). This was not significantly impacted by reducing the set of predictor variables to the 10 most important variables identified in the GBM model (free-text and coded cardiomyopathy, ischaemic heart disease and atrial fibrillation, digoxin, mineralocorticoid receptor antagonists, and combinations of renin-angiotensin system inhibitors and beta-blockers with diuretics). This optimized query set was enough to identify 86% (n = 461/538) of GPs' self-assessed HF population with a 33% reduction (n = 1537/4678) in screening caseload.

Conclusions: Diagnostic coding of HF in primary care health records is inaccurate with a high degree of under-registration and over-registration. An optimized query set enabled identification of more than 80% of GPs' self-assessed HF population.

Keywords: Audit and feedback; Chronic heart failure; Electronic health records; Primary care; Screening.

Figure 1

Study flowchart describing the audit process in four stages. (1) Identification of the patient population aged 40 years or older. (2) Identification of possible heart failure (HF) patients through an extended audit in the electronic health record. (3) Identification of patients with registered HF before GP assessment. (4) GP assessment of the list generated in Step 2 as HF/no HF. GP, general practitioner.

Figure 2

Alluvial diagram of the classification flow of patients who had registered HF before or after general practitioner assessment. HF, heart failure.

Figure 3

Bar chart illustrating the frequency of positive responses for each query for the entire population identified in the audit (n = 4679). Green = patients assessed as HF by their GP, orange = patients assessed as non‐HF by their GP. ACEi, angiotensin‐converting enzyme inhibitor; ARB, angiotensin receptor blocker; BB, beta‐blocker; C, coded query; FT, free‐text query; GP, general practitioner; HF, heart failure.

Figure 4

Variable importance score for each query for two different modelling strategies, scaled on the most important variable. ACEi, angiotensin‐converting enzyme inhibitor; ARB, angiotensin receptor blocker; BB, beta‐blocker; C, coded query; FT, free‐text query; GBM, gradient boosted machine; GP, general practitioner; HF, heart failure.

Figure 5

Comparison of sensitivity and specificity in a receiver‐operating characteristic curve for the identification of heart failure patients using four different modelling strategies. The labels indicate the sensitivity and specificity at the optimal cut‐off for each strategy. GBM, gradient boosted machine.

Figure 6

Precision–recall curve illustrating the benefit of adding queries with a logical OR operator in the entire data set. The y‐axis indicates precision or positive predicative value (PPV). The x‐axis indicates recall or true positive ratio (TPR). Integers in the graph depict the number of queries combined. The size of the circles expresses the total number of identified patients; the green inner circle expresses the proportion of OSCAR‐HF patients. ACEi, angiotensin‐converting enzyme inhibitors; AF, atrial fibrillation; ARB, angiotensin receptor inhibitors; BB, beta‐blockers; C, coded query; CMP, cardiomyopathy; FT, free‐text query; HF, heart failure; IHD, ischaemic heart disease; MRA, mineralocorticoid receptor antagonists.