Utilizing a domain-specific large language model for LI-RADS v2018 categorization of free-text MRI reports: a feasibility study

Abstract

Objective: To develop a domain-specific large language model (LLM) for LI-RADS v2018 categorization of hepatic observations based on free-text descriptions extracted from MRI reports.

Material and methods: This retrospective study included 291 small liver observations, divided into training (n = 141), validation (n = 30), and test (n = 120) datasets. Of these, 120 were fictitious, and 171 were extracted from 175 MRI reports from a single institution. The algorithm's performance was compared to two independent radiologists and one hepatologist in a human replacement scenario, and considering two combined strategies (double reading with arbitration and triage). Agreement on LI-RADS category and dichotomic malignancy (LR-4, LR-5, and LR-M) were estimated using linear-weighted κ statistics and Cohen's κ, respectively. Sensitivity and specificity for LR-5 were calculated. The consensus agreement of three other radiologists served as the ground truth.

Results: The model showed moderate agreement against the ground truth for both LI-RADS categorization (κ = 0.54 [95% CI: 0.42-0.65]) and the dichotomized approach (κ = 0.58 [95% CI: 0.42-0.73]). Sensitivity and specificity for LR-5 were 0.76 (95% CI: 0.69-0.86) and 0.96 (95% CI: 0.91-1.00), respectively. When the chatbot was used as a triage tool, performance improved for LI-RADS categorization (κ = 0.86/0.87 for the two independent radiologists and κ = 0.76 for the hepatologist), dichotomized malignancy (κ = 0.94/0.91 and κ = 0.87) and LR-5 identification (1.00/0.98 and 0.85 sensitivity, 0.96/0.92 and 0.92 specificity), with no statistical significance compared to the human readers' individual performance. Through this strategy, the workload decreased by 45%.

Conclusion: LI-RADS v2018 categorization from unlabelled MRI reports is feasible using our LLM, and it enhances the efficiency of data curation.

Critical relevance statement: Our proof-of-concept study provides novel insights into the potential applications of LLMs, offering a real-world example of how these tools could be integrated into a local workflow to optimize data curation for research purposes.

Key points: Automatic LI-RADS categorization from free-text reports would be beneficial to workflow and data mining. LiverAI, a GPT-4-based model, supported various strategies improving data curation efficiency by up to 60%. LLMs can integrate into workflows, significantly reducing radiologists' workload.

Keywords: Hepatocellular carcinoma; Natural language processing; Radiology; Report; Standardization.

Fig. 1

Flowchart of study design

Fig. 2

Chatbot interface of the domain-specific LLM (LiverAI), showing an example of a free-text liver observation description correctly categorized. Both the chatbot interface and the report have been translated from Spanish to English for understandability reasons. LI-RADS, liver imaging reporting and data system

Fig. 3

Workflow and performance analysis of the domain-specific LLM (LiverAI). LI-RADS, liver imaging reporting and data system; R1 and R2, independent radiologist 1 and 2, respectively; H, hepatologist

Fig. 4

The clinical scenarios evaluated to assess the optimal integration of the domain-specific LLM (LiverAI) for LI-RADS categorization of liver observations described in free-text MRI reports. R1 and R2, independent radiologists 1 and 2, respectively. * The double reading strategy was repeated considering both the combination of R1 and LiverAI, with posterior arbitration by R2; and the combination of R2 and LiverAI, with arbitration by R1

Fig. 5

Inter-reader agreement across LI-RADS categories among all human readers. R, readers

Fig. 6

Assessment by LiverAI is compared to the consensus radiologic assessment across all LI-RADS categories (a) and dichotomized malignancy (b). RCA, radiologic consensus assessment