Automated transformation of unstructured cardiovascular diagnostic reports into structured datasets using sequentially deployed large language models

Abstract

Aims: Rich data in cardiovascular diagnostic testing are often sequestered in unstructured reports, limiting their use.

Methods and results: We sequentially deployed generative and interpretative open-source large language models (LLMs; Llama2-70b, Llama2-13b). Using Llama2-70b, we generated varying formats of transthoracic echocardiogram (TTE) reports from 3000 real-world reports with paired structured elements. Using prompt-based supervised training, we fine-tuned Llama2-13b using sequentially larger batches of generated TTE reports as inputs, to extract data across 18 clinically-relevant echocardiographic fields. We evaluated the fine-tuned model, HeartDX-LM, on distinct datasets: (i) different time periods and formats at Yale New Haven Health System (YNHHS), (ii) Medical Information Mart for Intensive Care (MIMIC) III, and (iii) MIMIC IV. We used accuracy and Cohen's kappa as evaluation metrics and have publicly released the HeartDX-LM model. HeartDX-LM was trained on 2000 synthetic reports with varying formats and paired structured labels. We identified a lower threshold of 500 unstructured reports-structured data pairs required for fine-tuning to achieve consistent performance. At YNHHS, HeartDX-LM accurately extracted 69 144 of 70 032 values (98.7%) across 18 fields in the contemporary test set where paired structured data were available. In 100 older YNHHS reports, HeartDX-LM achieved 87.1% accuracy against expert annotations. In external validation sets from MIMIC-III and MIMIC-IV, HeartDX-LM correctly extracted 615 of 707 available values (87.9%) and 201 of 220 available values (91.3%), from 100 random, expert-annotated reports from each set.

Conclusion: We developed and validated a novel approach using paired large and moderate-sized LLMs to transform free-text echocardiographic reports into tabular datasets.

Keywords: Echocardiograph; Large language models; Machine learning; Natural language processing.

Graphical Abstract

Figure 1

Model development approach and study design. YNHHS, Yale New Haven Health System; MIMIC, Medical Information Mart for Intensive Care.

Figure 2

Accuracy of HeartDX-LM for label extraction across the four datasets. AVA Cont VTI, aortic valve area calculated by velocity time integral; AVA Index, aortic valve area index; AV Mn Grad, aortic valve mean gradient; AIPHT, aortic insufficiency pressure half-time; AV Pk Vel, aortic valve peak velocity; AV Regurgitation, aortic valve regurgitation; AV Stenosis, aortic valve stenosis; AV Structure, aortic valve structure; GLS, global longitudinal strain; IVSd, interventricular septum thickness; LV Diastolic Function, left ventricular diastolic function; LVOT Pk Grad, left ventricular outflow tract peak gradient; LVOT Pk Vel, left ventricular outflow tract peak velocity; LV Wall Thickness, left ventricular wall thickness; MV Regurgitation, mitral valve regurgitation; MV Stenosis, mitral valve stenosis; MV Structure, mitral valve structure.

Figure 3

Performance of models fine-tuned with varying number of paired unstructured reports and structured tables for tabulation of clinical variables from unstructured reports. AVA Cont VTI, aortic valve area calculated by velocity time integral; AVA Index, aortic valve area index; AV Mn Grad, aortic valve mean gradient; AIPHT, aortic insufficiency pressure half-time; AV Pk Vel, aortic valve peak velocity; AV Regurgitation, aortic valve regurgitation; AV Stenosis, aortic valve stenosis; AV Structure, aortic valve structure; GLS, global longitudinal strain; IVSd, interventricular septum thickness; LV Diastolic Function, left ventricular diastolic function; LVOT Pk Grad, left ventricular outflow tract peak gradient; LVOT Pk Vel, left ventricular outflow tract peak velocity; LV Wall Thickness, left ventricular wall thickness; MV Regurgitation, mitral valve regurgitation; MV Stenosis, mitral valve stenosis; MV Structure, mitral valve structure.