Comparing artificial intelligence- vs clinician-authored summaries of simulated primary care electronic health records

Abstract

Objective: To compare clinical summaries generated from simulated patient primary care electronic health records (EHRs) by GPT-4, to summaries generated by clinicians on multiple domains of quality including utility, concision, accuracy, and bias.

Materials and methods: Seven primary care physicians generated 70 simulated patient EHR notes, each representing 10 patient contacts with the practice over at least 2 years. Each record was summarized by a different clinician and by GPT-4. artificial intelligence (AI)- and clinician-authored summaries were rated blind by clinicians according to 8 domains of quality and an overall rating.

Results: The median time taken for a clinician to read through and assimilate the information in the EHRs before summarizing, was 7 minutes. Clinicians rated clinician-authored summaries higher than AI-authored summaries overall (7.39 vs 7.00 out of 10; P = .02), but with greater variability in clinician-authored summary ratings. AI and clinician-authored summaries had similar accuracy and AI-authored summaries were less likely to omit important information and more likely to use patient-friendly language.

Discussion: Although AI-authored summaries were rated slightly lower overall compared with clinician-authored summaries, they demonstrated similar accuracy and greater consistency. This demonstrates potential applications for generating summaries in primary care, particularly given the substantial time taken for clinicians to undertake this work.

Conclusion: The results suggest the feasibility, utility and acceptability of using AI-authored summaries to integrate into EHRs to support clinicians in primary care. AI summarization tools have the potential to improve healthcare productivity, including by enabling clinicians to spend more time on direct patient care.

Keywords: electronic health records; generative AI; health informatics; large language models; primary care.

Figure 1.

Schematic of study design.

Figure 2.

Example template and first (of 10) free text entries for a simulated patient record.

Figure 3.

Qualities of summaries rated by clinicians on 5-point Likert scale.

Figure 4.

Pairwise distances between the 3 ratings assigned by clinicians for clinician and AI-authored summaries.

Figure 5.

Combined figure of all responses (cross-hatched bars are clinician-authored, plain are AI-authored).