How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record

Abstract

Purpose: Despite widespread reliance on chart abstraction for quality measurement, concerns persist about its reliability and validity. We prospectively evaluated the validity of chart abstraction by directly comparing it with the gold standard of reports by standardized patients.

Subjects and methods: Twenty randomly selected general internal medicine residents and attending faculty physicians at the primary care clinics of two Veterans Affairs Medical Centers blindly evaluated and treated actor-patients (standardized patients) who had one of four common diseases: diabetes, chronic obstructive pulmonary disease, coronary artery disease, or low back pain. Charts from the visits were abstracted using explicit quality criteria; standardized patients completed a checklist containing the same criteria. For each physician, quality was measured for two different cases of the four conditions (a total of 160 physician-patient encounters). We compared chart abstraction with standardized-patient reports for four aspects of the encounter: taking the history, examining the patient, making the diagnosis, and prescribing appropriate treatment. The sensitivity and specificity of chart abstraction were calculated.

Results: The mean (+/- SD) chart abstraction score was 54% +/- 9%, substantially less than the mean score on the standardized-patient checklist of 68% +/- 9% (P <0.001). This finding was similar for all four conditions and at both sites. "False positives"-chart-recorded necessary care actions not reported by the standardized patients-resulted in a specificity of only 81%. The overall sensitivity of chart abstraction for necessary care was only 70%.

Conclusions: Chart abstraction underestimates the quality of care for common outpatient general medical conditions when compared with standardized-patient reports. The medical record is neither sensitive nor specific. Quality measurements derived from chart abstraction may have important shortcomings, particularly as the basis for drawing policy conclusions or making management decisions.