Automatic structuring of radiology reports with on-premise open-source large language models

Abstract

Objectives: Structured reporting enhances comparability, readability, and content detail. Large language models (LLMs) could convert free text into structured data without disrupting radiologists' reporting workflow. This study evaluated an on-premise, privacy-preserving LLM for automatically structuring free-text radiology reports.

Materials and methods: We developed an approach to controlling the LLM output, ensuring the validity and completeness of structured reports produced by a locally hosted Llama-2-70B-chat model. A dataset with de-identified narrative chest radiograph (CXR) reports was compiled retrospectively. It included 202 English reports from a publicly available MIMIC-CXR dataset and 197 German reports from our university hospital. Senior radiologist prepared a detailed, fully structured reporting template with 48 question-answer pairs. All reports were independently structured by the LLM and two human readers. Bayesian inference (Markov chain Monte Carlo sampling) was used to estimate the distributions of Matthews correlation coefficient (MCC), with [-0.05, 0.05] as the region of practical equivalence (ROPE).

Results: The LLM generated valid structured reports in all cases, achieving an average MCC of 0.75 (94% HDI: 0.70-0.80) and F1 score of 0.70 (0.70-0.80) for English, and 0.66 (0.62-0.70) and 0.68 (0.64-0.72) for German reports, respectively. The MCC differences between LLM and humans were within ROPE for both languages: 0.01 (-0.05 to 0.07), 0.01 (-0.05 to 0.07) for English, and -0.01 (-0.07 to 0.05), 0.00 (-0.06 to 0.06) for German, indicating approximately comparable performance.

Conclusion: Locally hosted, open-source LLMs can automatically structure free-text radiology reports with approximately human accuracy. However, the understanding of semantics varied across languages and imaging findings.

Key points: Question Why has structured reporting not been widely adopted in radiology despite clear benefits and how can we improve this? Findings A locally hosted large language model successfully structured narrative reports, showing variation between languages and findings. Critical relevance Structured reporting provides many benefits, but its integration into the clinical routine is limited. Automating the extraction of structured information from radiology reports enables the capture of structured data while allowing the radiologist to maintain their reporting workflow.

Keywords: Chest radiography; Large language models; Structured reporting.

Fig. 1

Reporting template. The template consists of 48 question-answer pairs and includes questions with a binary answer (possible answers: finding present or absent, hidden for clarity), marked with question marks, and questions with specified answer options, marked with a colon (possible answers provided after the colon). The template includes nested questions, answered only if the parent finding is present

Fig. 2

Study overview. Chest radiography reports from two sources were analyzed: MIMIC-CXR (English) and UH (German). The open-access Llama-2-70B model was used to extract structured elements from free-text radiology reports. The results of the automated structuring were compared with human readers. Llama-2-70B image was generated using GPT-4 through https://chat.openai.com/ on 11.11.2023. MIMIC-CXR, MIMIC chest X-ray cohort; UH, University Hospital cohort

Fig. 3

Distribution of Matthews correlation coefficient (MCC) for Llama-2-70B. The kernel density plot presents the posterior distribution of the MCC with the 94% highest density interval. Rhomboid markers denote quartiles. The red distributions represent the cumulative MCC across all findings in a template section. MIMIC-CXR, MIMIC chest X-ray cohort; UH, University Hospital cohort

Fig. 4

Distribution of pairwise differences in Matthews correlation coefficient (MCC). The kernel density plot shows the posterior distribution of the MCC pairwise differences with the 94% highest density interval. Rhomboid markers denote quartiles. The green vertical shaded area is the region of practical equivalence (−0.05, 0.05). The red distributions represent the cumulative differences across all labels. MIMIC-CXR, MIMIC chest X-ray cohort; UH, University Hospital cohort