Ontology-guided machine learning outperforms zero-shot foundation models for cardiac ultrasound text reports

Abstract

Big data can revolutionize research and quality improvement for cardiac ultrasound. Text reports are a critical part of such analyses. Cardiac ultrasound reports include structured and free text and vary across institutions, hampering attempts to mine text for useful insights. Natural language processing (NLP) can help and includes both statistical- and large language model based techniques. We tested whether we could use NLP to map cardiac ultrasound text to a three-level hierarchical ontology. We used statistical machine learning (EchoMap) and zero-shot inference using GPT. We tested eight datasets from 24 different institutions and compared both methods against clinician-scored ground truth. Despite all adhering to clinical guidelines, institutions differed in their structured reporting. EchoMap performed best with validation accuracy of 98% for the first ontology level, 93% for first and second levels, and 79% for all three. EchoMap retained performance across external test datasets and could extrapolate to examples not included in training. EchoMap's accuracy was comparable to zero-shot GPT at the first level of the ontology and outperformed GPT at second and third levels. We show that statistical machine learning can map text to structured ontology and may be especially useful for small, specialized text datasets.

Keywords: Echocardiography report; Large language models; Machine learning; Natural language processing; Ontology.

Fig. 1

Workflow for the three machine learning approaches evaluated. Data (structured dictionaries and free text from echo reports) were preprocessed, then passed to each of three model types: (A) Hierarchical random forest statistical machine learning model, which included additional engineered features and used each level’s prediction to inform the subsequent level, (B) zero-shot GPT making independent predictions per level of ontology, (C) zero-shot GPT making multi-class prediction. GPT, generative pre-trained transformer; RF, random forest; UMLS, unified medical language system; L1, L2, L3, Level 1, Level 2, Level 3, respectively.

Fig. 2

Correctness by ontology level, by dataset. Validation (UCSF) and test set (outside hospitals) performance for each of three model architectures: (A) Echomap, (B) Zero-shot GPT, (C) multi-class zero-shot GPT. UCSF, University of California, San Francisco. PITT, University of Pittsburgh; IU, Indiana University; UCSF-OSH, outside hospital reports in the UCSF system; UAZ, University of Arizona; UCSF-FREE, free-text sentences from UCSF reports; UPENN, University of Pennsylvania; UW, University of Washington.

Fig. 3

Aggregate performance across all datasets evaluated, by each mapping model and by ontology level. Box plots represent performance of all eight validation and test datasets in order to illustrate differences among mapping models.