Generative Pre-trained Transformer 4 analysis of cardiovascular magnetic resonance reports in suspected myocarditis: A multicenter study

Abstract

Background: Diagnosing myocarditis relies on multimodal data, including cardiovascular magnetic resonance (CMR), clinical symptoms, and blood values. The correct interpretation and integration of CMR findings require radiological expertise and knowledge. We aimed to investigate the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model, for report-based medical decision-making in the context of cardiac MRI for suspected myocarditis.

Methods: This retrospective study includes CMR reports from 396 patients with suspected myocarditis and eight centers, respectively. CMR reports and patient data including blood values, age, and further clinical information were provided to GPT-4 and radiologists with 1 (resident 1), 2 (resident 2), and 4 years (resident 3) of experience in CMR and knowledge of the 2018 Lake Louise Criteria. The final impression of the report regarding the radiological assessment of whether myocarditis is present or not was not provided. The performance of Generative pre-trained transformer 4 (GPT-4) and the human readers were compared to a consensus reading (two board-certified radiologists with 8 and 10 years of experience in CMR). Sensitivity, specificity, and accuracy were calculated.

Results: GPT-4 yielded an accuracy of 83%, sensitivity of 90%, and specificity of 78%, which was comparable to the physician with 1 year of experience (R1: 86%, 90%, 84%, p = 0.14) and lower than that of more experienced physicians (R2: 89%, 86%, 91%, p = 0.007 and R3: 91%, 85%, 96%, p < 0.001). GPT-4 and human readers showed a higher diagnostic performance when results from T1- and T2-mapping sequences were part of the reports, for residents 1 and 3 with statistical significance (p = 0.004 and p = 0.02, respectively).

Conclusion: GPT-4 yielded good accuracy for diagnosing myocarditis based on CMR reports in a large dataset from multiple centers and therefore holds the potential to serve as a diagnostic decision-supporting tool in this capacity, particularly for less experienced physicians. Further studies are required to explore the full potential and elucidate educational aspects of the integration of large language models in medical decision-making.

Keywords: Artificial intelligence; Cardiovascular magnetic resonance; Generative Pre-trained Transformer 4; Large language models; Myocarditis.

Graphical abstract

Fig. 1

Exemplary styles of reports being included in this study: free text with T1/T2-mapping, free text without T1/T2-mapping, structured report with T1- and T2-mapping, and structured report without T1/T2-mapping. LV left ventricular, EDV end-diastolic volume, ESV end-systolic volume, EF ejection fraction, CO cardiac output, CI cardiac index, HR heart rate, CT computed tomography, LV EDD left ventricular end-diastolic diameter, BSA body surface area, ED end-diastole, LGE late gadolinium enhancement, GPT-4 Generative Pre-trained Transformer 4.

Fig. 2

Workflow of the study design. The reference standard was established by the diagnosis of myocarditis based on the assessment of two board-certified radiologists with 8 and 10 years of experience in cardiovascular imaging, respectively. GPT-4 Generative Pre-trained Transformer 4.

Fig. 3

Confusion matrices for performance of GPT-4, residents 1, 2, and 3 compared to the reference standard. GPT-4: Generative Pre-trained Transformer 4.

Fig. 4

Proofreading examples by GPT-4 based on the given clinical data, laboratory values, and the radiology report compared to the assessment of the human readers. CRP C-reactive protein, CK creatine kinase, CK-MB creatine kinase-MB, Hs-Trop high sensitive troponin. IVSD interventricular septum thickness. LV left ventricular, EDV end-diastolic volume, EF ejection fraction, BSA body surface area, GPT-4 Generative Pre-trained Transformer 4, CMR cardiovascular magnetic resonance .