Empowering PET imaging reporting with retrieval-augmented large language models and reading reports database: a pilot single center study

Abstract

Purpose: The potential of Large Language Models (LLMs) in enhancing a variety of natural language tasks in clinical fields includes medical imaging reporting. This pilot study examines the efficacy of a retrieval-augmented generation (RAG) LLM system considering zero-shot learning capability of LLMs, integrated with a comprehensive database of PET reading reports, in improving reference to prior reports and decision making.

Methods: We developed a custom LLM framework with retrieval capabilities, leveraging a database of over 10 years of PET imaging reports from a single center. The system uses vector space embedding to facilitate similarity-based retrieval. Queries prompt the system to generate context-based answers and identify similar cases or differential diagnoses. From routine clinical PET readings, experienced nuclear medicine physicians evaluated the performance of system in terms of the relevance of queried similar cases and the appropriateness score of suggested potential diagnoses.

Results: The system efficiently organized embedded vectors from PET reports, showing that imaging reports were accurately clustered within the embedded vector space according to the diagnosis or PET study type. Based on this system, a proof-of-concept chatbot was developed and showed the framework's potential in referencing reports of previous similar cases and identifying exemplary cases for various purposes. From routine clinical PET readings, 84.1% of the cases retrieved relevant similar cases, as agreed upon by all three readers. Using the RAG system, the appropriateness score of the suggested potential diagnoses was significantly better than that of the LLM without RAG. Additionally, it demonstrated the capability to offer differential diagnoses, leveraging the vast database to enhance the completeness and precision of generated reports.

Conclusion: The integration of RAG LLM with a large database of PET imaging reports suggests the potential to support clinical practice of nuclear medicine imaging reading by various tasks of AI including finding similar cases and deriving potential diagnoses from them. This study underscores the potential of advanced AI tools in transforming medical imaging reporting practices.

Keywords: Artificial intelligence; Large language model; PET reports; Retrieval-augmented generation.

Fig. 1

Workflow of the Chatbot System for Querying PET Imaging Reading Reports. The overall workflow of the proof-of-concept system designed for efficient querying of reading reports from a substantial dataset is illustrated. The system integrates the Retrieval-Augmented Generation (RAG) model with advanced language model technologies, natural language processing, and information retrieval techniques. The workflow demonstrates the process from user query input through to the delivery of the relevant reading report, showcasing the operational framework and interaction with different sources of reading reports

Fig. 2

Visualization of PET Imaging Report Embeddings Using t-SNE. (A) t-SNE plot illustrates PET imaging report embeddings from 118,107 patients, totaling 211,813 cases. Each point on the plot represents a unique report, with a selected case highlighted in red to show an example of an original report. (B) t-SNE plots showcases the clustering efficacy of the embeddings, highlighting how reports containing key diagnostic terms like ‘lung cancer’, ‘breast cancer’, ‘lymphoma’, and specific types of exams such as ‘C-11 methionine PET’ and ‘Ga-68 PSMA-11 PET’ form distinct clusters. These clusters indicate the embeddings’ capability to reflect the similarity among cases, demonstrating the potential of this method in facilitating the identification and visualization of related PET imaging reports

Fig. 3

Examples of Chatbot Responses to Queries. (A) An example case displays an instance of the chatbot’s capability to accurately identify and present relevant cases in response to a user query about breast cancer with metastasis to internal mammary lymph nodes. It highlights the capacity to navigate a vast database of previous reading reports to identify relevant cases. (B) An example of the utility of system in generating differential diagnoses is displayed. This is demonstrated through the chatbot’s response to a query, where it offers a detailed list of potential diagnoses along with reference identifiers. As an example, by employing identifiers within the PACS system (in this example, we used deidentified information), prior imaging cases could be referenced for understanding cases and supporting decision making

Fig. 4

Evaluation of Appropriateness Scores by Nuclear Medicine Physicians. (A) The appropriateness of querying similar cases was assessed. Using a conclusion text to generate the prompt “find similar reports and summarize it,” the system retrieved results. For specific reports, 16 out of 19 (84.2%) were appropriately identified, with all three readers rating these as better than ‘Fair’ in relevance. (B) The appropriateness of potential diagnoses for specific findings was evaluated. Using specific finding texts to generate prompts for suggesting potential diagnoses, the responses of system were assessed. Medical relevance and appropriateness of the suggested potential diagnoses were evaluated by readers. The system without RAG was also assessed, and the performance of the LLM with and without RAG was represented as a heatmap. The results indicated that the LLM with RAG showed significantly better appropriateness scores (p < 0.05). (C) The ROUGE-L F-score was used to quantitatively evaluate the alignment between generated conclusions and reference conclusion texts from finding descriptions. The RAG framework demonstrated significantly higher scores compared to the LLM without RAG (0.16 ± 0.08 vs. 0.07 ± 0.03, p < 0.001)