Generating colloquial radiology reports with large language models

Abstract

Objectives: Patients are increasingly being given direct access to their medical records. However, radiology reports are written for clinicians and typically contain medical jargon, which can be confusing. One solution is for radiologists to provide a "colloquial" version that is accessible to the layperson. Because manually generating these colloquial translations would represent a significant burden for radiologists, a way to automatically produce accurate, accessible patient-facing reports is desired. We propose a novel method to produce colloquial translations of radiology reports by providing specialized prompts to a large language model (LLM).

Materials and methods: Our method automatically extracts and defines medical terms and includes their definitions in the LLM prompt. Using our method and a naive strategy, translations were generated at 4 different reading levels for 100 de-identified neuroradiology reports from an academic medical center. Translations were evaluated by a panel of radiologists for accuracy, likability, harm potential, and readability.

Results: Our approach translated the Findings and Impression sections at the 8th-grade level with accuracies of 88% and 93%, respectively. Across all grade levels, our approach was 20% more accurate than the baseline method. Overall, translations were more readable than the original reports, as evaluated using standard readability indices.

Conclusion: We find that our translations at the eighth-grade level strike an optimal balance between accuracy and readability. Notably, this corresponds to nationally recognized recommendations for patient-facing health communication. We believe that using this approach to draft patient-accessible reports will benefit patients without significantly increasing the burden on radiologists.

Keywords: large language model; machine learning; natural language processing; prompt engineering; radiology.

Figure 1.

Our translation pipeline: medical knowledge (MK)-based prompting.

Figure 2.

Illustration of the steps involved for 1 example report.

Figure 3.

(A and B): Accuracy of translations generated by each prompting method at 4 different education levels. (C and D): Physician likability of translations generated by each prompting method at 4 different education levels. Higher is better. (E and F): Readability index of translations generated by each prompting method at 4 different education levels. Lower readability index corresponds to simpler language. (G and H): Harm potential of translations generated by each prompting method at 4 different education levels. Lower is better. In all panels, asterisk indicates statistical significance for MK + Indications model vs baseline (P < .05).

Figure 4.

Example of attribution when the translation is accurate. The attribution is shown by color coding the sentences, for example, the text in the translation is attributed to the text in the original report. Note that there is text in the original report that is not used for attribution, meaning that it is not present in the translation.

Figure 5.

Example of attribution when there is hallucination in the translation. The attribution output for each of the sentences in the translation says “NOT FOUND.”