Impact of Unannounced Standardized Patient Audit and Feedback on Care, Documentation, and Costs: an Experiment and Claims Analysis

Abstract

Background: Meaningful variations in physician performance are not always discernible from the medical record.

Objective: We used unannounced standardized patients to measure and provide feedback on care quality and fidelity of documentation, and examined downstream effects on reimbursement claims.

Design: Static group pre-post comparison study conducted between 2017 and 2019.

Setting: Fourteen New Jersey primary care practice groups (22 practices) enrolled in Horizon BCBS's value-based program received the intervention. For claims analyses, we identified 14 additional comparison practice groups matched on county, practice size, and claims activity.

Participants: Fifty-nine of 64 providers volunteered to participate.

Intervention: Unannounced standardized patients (USPs) made 217 visits portraying patients with 1-2 focal conditions (diabetes, depression, back pain, smoking, or preventive cancer screening). After two baseline visits to a provider, we delivered feedback and conducted two follow-up visits.

Measurements: USP-completed checklists of guideline-based provider care behaviors, visit audio recordings, and provider notes were used to measure behaviors performed and documentation errors pre- and post-feedback. We also compared changes in 3-month office-based claims by actual patients between the intervention and comparison practice groups before and after feedback.

Results: Expected clinical behaviors increased from 46% to 56% (OR = 1.53, 95% CI 1.29-1.83, p < 0.0001), with significant improvements in smoking cessation, back pain, and depression screening. Providers were less likely to document unperformed tasks after (16%) than before feedback (18%; OR = 0.74, 95% CI 0.62 to 0.90, p = 0.002). Actual claim costs increased significantly less in the study than comparison group for diabetes and depression but significantly more for smoking cessation, cancer screening, and low back pain.

Limitations: Self-selection of participating practices and lack of access to prescription claims.

Conclusion: Direct observation of care identifies hidden deficits in practice and documentation, and with feedback can improve both, with concomitant effects on costs.

Keywords: cost of care; directly observed care; health services research; quality improvement; unannounced standardized patients.

Figure 1

Observed frequency of targeted behaviors recorded by USPs (“Heard”) vs. entered in the medical record (“Documented”) in pre- and post-feedback visits. Error bars indicate the upper limits of the 95% confidence intervals around the proportions.

Figure 2

Predicted average per-visit claims by group, time period, and focal condition.