Data extraction from free-text stroke CT reports using GPT-4o and Llama-3.3-70B: the impact of annotation guidelines

Abstract

Background: To evaluate the impact of an annotation guideline on the performance of large language models (LLMs) in extracting data from stroke computed tomography (CT) reports.

Methods: The performance of GPT-4o and Llama-3.3-70B in extracting ten imaging findings from stroke CT reports was assessed in two datasets from a single academic stroke center. Dataset A (n = 200) was a stratified cohort including various pathological findings, whereas dataset B (n = 100) was a consecutive cohort. Initially, an annotation guideline providing clear data extraction instructions was designed based on a review of cases with inter-annotator disagreements in dataset A. For each LLM, data extraction was performed under two conditions: with the annotation guideline included in the prompt and without it.

Results: GPT-4o consistently demonstrated superior performance over Llama-3.3-70B under identical conditions, with micro-averaged precision ranging from 0.83 to 0.95 for GPT-4o and from 0.65 to 0.86 for Llama-3.3-70B. Across both models and both datasets, incorporating the annotation guideline into the LLM input resulted in higher precision rates, while recall rates largely remained stable. In dataset B, the precision of GPT-4o and Llama-3-70B improved from 0.83 to 0.95 and from 0.87 to 0.94, respectively. Overall classification performance with and without the annotation guideline was significantly different in five out of six conditions.

Conclusion: GPT-4o and Llama-3.3-70B show promising performance in extracting imaging findings from stroke CT reports, although GPT-4o steadily outperformed Llama-3.3-70B. We also provide evidence that well-defined annotation guidelines can enhance LLM data extraction accuracy.

Relevance statement: Annotation guidelines can improve the accuracy of LLMs in extracting findings from radiological reports, potentially optimizing data extraction for specific downstream applications.

Key points: LLMs have utility in data extraction from radiology reports, but the role of annotation guidelines remains underexplored. Data extraction accuracy from stroke CT reports by GPT-4o and Llama-3.3-70B improved when well-defined annotation guidelines were incorporated into the model prompt. Well-defined annotation guidelines can improve the accuracy of LLMs in extracting imaging findings from radiological reports.

Keywords: Artificial intelligence; Information storage and retrieval; Large language models; Stroke; Tomography (x-ray computed).

Fig. 1

Study design. Initially, two raters annotated dataset A using a preliminary annotation guideline with a few general instructions. Based on the guideline deficiencies uncovered based on a review of cases with inter-rater disagreement, an addendum was appended to the original document, forming the final annotation guideline. The data extraction performance of GPT-4o and Llama-3.3-70B with and without the annotation guideline was evaluated in dataset A and additionally in another dataset (dataset B) that was not used to formulate the annotation guideline. At the bottom, a fictional CT report in English is shown along with the data parameters extracted from it in JSON format to illustrate the methodology

Fig. 2

Data extraction performance of GPT-4o and Llama-3.3-70B across all parameters. Metrics for GPT-4o queries with a temperature setting of 1 were calculated based on the mode across three repetitions, whereas the remaining queries were run only once (with a temperature setting of 0.0). Error bars indicate 95% confidence intervals. The 1.7% (34/2,000) and 1.4% (14/1,000) of data points were excluded from dataset A and dataset B, each, as the report text indicated diagnostic uncertainty without a clear positive or negative tendency (expressions such as “possible”, “DDx”). Precision: positive predictive value. Recall: sensitivity. F1-Score: harmonic mean of precision and recall