Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2026 Feb 26.
doi: 10.1038/s41746-026-02494-9. Online ahead of print.

Vision-Enabled AI scribes reduce omissions in clinical conversations: evidence from simulated medication histories

Bradley D Menz  1   2 , Nicholas L Scarfo  3 , Natansh D Modi  4 , Erik Cornelisse  5   3 , Lee X Li  5 , Jin Quan Eugene Tan  5   3 , Jimit Gandhi  4   6 , Dorsa Maher  5 , Dib Kousa  3 , Kezia Daniel  6 , Vidya Menon  3 , Stephen Bacchi  7 , Ross A McKinnon  5 , Michael D Wiese  4 , Andrew Rowland  5 , Michael J Sorich  5 , Ashley M Hopkins  5
Affiliations
Free article

Vision-Enabled AI scribes reduce omissions in clinical conversations: evidence from simulated medication histories

Bradley D Menz et al. NPJ Digit Med. .
Free article

Abstract

Most ambient AI medical scribes process audio only, omitting clinically important visual details. We developed a vision-enabled AI scribe using Google's Gemini model and Ray-Ban Meta smart glasses to document medication histories-a task requiring both audio and visual input. Ten clinical pharmacists video-recorded 110 simulated medication history interviews. Following iterative prompt engineering on 10 training recordings, the scribe was evaluated on 100 test recordings (2160 data points) across patient details and medication-specific fields. The vision-enabled scribe achieved 98% overall accuracy (2114/2,160 data points), ranging from 96% for patient details to 99% for dosing directions and indication. Video input significantly outperformed audio-only processing (98% vs 81%, P < 0.001), primarily through reduced omissions (10 vs 358 errors). Vision-enabled AI scribes substantially improved documentation accuracy for tasks requiring visual input, demonstrating potential to markedly reduce omission errors in clinical documentation.

PubMed Disclaimer

Conflict of interest statement

Competing interests: A.M.H. is a recipient of investigator-initiated funding for research outside the scope of the current study from Boehringer Ingelheim. A.R. and M.J.S. are recipients of investigator-initiated funding for research outside the scope of the current study from AstraZeneca, Boehringer Ingelheim, Pfizer and Takeda. A.R. is a recipient of speaker fees from Boehringer Ingelheim and Genentech. The author team have no other potential conflicts of interest with respect to this research and/or publication to declare.

References

    1. Tierney, A. A. et al. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catalyst 5, CAT.23.0404 (2024).
    1. Sorich, M. J., Mangoni, A. A., Bacchi, S., Menz, B. D. & Hopkins, A. M. The triage and diagnostic accuracy of frontier large language models: updated comparison to physician performance. J. Med. Internet Res. 26, e67409 (2024).
    1. Menz, B. D. et al. Generative AI chatbots for reliable cancer information: evaluating web-search, multilingual, and reference capabilities of emerging large language models. Eur. J. Cancer 218, 115274 (2025).
    1. Shahnam, A. et al. Application of generative artificial intelligence for physician and patient oncology letters—AI-OncLetters. JCO Clin. Cancer Inform. e2400323 (2025).
    1. Zaretsky, J. et al. Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format. JAMA Netw. Open 7, e240357 (2024).

LinkOut - more resources