Multimodal Artificial Intelligence in Medicine

Abstract

Traditional medical artificial intelligence models that are approved for clinical use restrict themselves to single-modal data ( e.g ., images only), limiting their applicability in the complex, multimodal environment of medical diagnosis and treatment. Multimodal transformer models in health care can effectively process and interpret diverse data forms, such as text, images, and structured data. They have demonstrated impressive performance on standard benchmarks, like United States Medical Licensing Examination question banks, and continue to improve with scale. However, the adoption of these advanced artificial intelligence models is not without challenges. While multimodal deep learning models like transformers offer promising advancements in health care, their integration requires careful consideration of the accompanying ethical and environmental challenges.

Figure 1

Patient scenarios without AI, with single modal AI and with multimodal AI. AI, artificial intelligence.

Figure 2

A future multimodal transformer-based AKI alert system. This figure depicts a multimodal transformer-based system designed to predict AKI risk by integrating text, structured data (e.g., laboratory tests), images (e.g., chest x-ray), and time series data. These are converted into numerical vectors through specific embedding layers. Positional encoding adds order information to these vectors, which are processed by the attention mechanism. The heatmap shows how the model focuses on relevant data parts, using keys, queries, and values to compute attention scores. Transformed vectors pass through a feed forward neural network for pattern learning. The architecture includes multiple identical decoder blocks for iterative refinement. At the top, the classification task predicts AKI risk at 24, 48, and 168 hours, triggering alerts if risk thresholds are exceeded. Cr, creatinine; Gl, glucose, Na, sodium; x H, attention heads; x K, decoder blocks.