Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions

Abstract

With the advent of large language models (LLMs), the artificial intelligence revolution in medicine and radiology is now more tangible than ever. Every day, an increasingly large number of articles are published that utilize LLMs in radiology. To adopt and safely implement this new technology in the field, radiologists should be familiar with its key concepts, understand at least the technical basics, and be aware of the potential risks and ethical considerations that come with it. In this review article, the authors provide an overview of the LLMs that might be relevant to the radiology community and include a brief discussion of their short history, technical basics, ChatGPT, prompt engineering, potential applications in medicine and radiology, advantages, disadvantages and risks, ethical and regulatory considerations, and future directions.

Keywords: ChatGPT; Large language models; artificial intelligence; deep learning; natural language processing.

Figure 1

Number of publications about language models in medical publications (green line) and medical imaging (yellow line) including radiology and nuclear medicine. Search date, July 20^th, 2023; source, PubMed.

Figure 2

Technical developmental stages of language models.

Figure 3

Key concepts in LLMs. Tokenization is the process of splitting text into smaller units (i.e., tokens) that can be processed by language models. Embedding is the mathematical representation of data (e.g., vector representation of a word). The attention mechanism allows the models to focus on certain parts of the input data. Pre-training is the training of a model to be used for a lot of different tasks without re-training from scratch. Fine-tuning is the adjustment of models to achieve improved performance on domain-specific tasks. Reinforcement learning from human feedback is a machine learning approach based on reinforcement learning techniques along with human guidance. LLMs, large language models.

Figure 4

Architecture of transformers. The encoder and decoder are overly simplified in the figure. Both normally include attention mechanisms, feed-forward neural networks, residual connections, and the normalization layer. Transformers utilize multiple layers of encoders and decoders. Nx, number of layers of encoder and decoder parts.

Figure 5

Tokenization example. A 10-word sentence with one punctuation sign is tokenized to 14 tokens as shown in the upper panel. The bottom panel shows token identifiers unique to each token. Generated by OpenAI’s Tokenizer platform (https://platform.openai.com/tokenizer).