AI-enabled molecular phenotyping and prognostic predictions in lung cancer through multimodal clinical information integration

Abstract

Lung cancer remains the leading cause of cancer-related mortality worldwide. The need for cost-effective, non-invasive methods to detect specific gene mutations for targeted therapy and predict patient survival outcomes underscores the importance of advancing diagnostic and prognostic capabilities. Contemporary lung cancer diagnostic models often fail to integrate diverse patient data, leading to incomplete clinical assessments. To address these challenges, we propose LUCID, a multimodal data integration framework designed to predict epidermal growth factor receptor (EGFR) mutation status and survival outcomes in patients with lung cancer. Tailored for early-stage clinical assessment, LUCID leverages lung computed tomography (CT) images, chief complaints, laboratory test results, and demographic data to deliver comprehensive, non-invasive predictions. LUCID achieved strong performance in a retrospective cohort of 5,175 patients, with areas under the receiver operating characteristic curve (AUCs) ranging from 0.851 to 0.881 for EGFR mutation prediction and from 0.821 to 0.912 for survival time prediction. The model also demonstrated robustness across external validation cohorts and in scenarios with missing modalities.

Keywords: EGFR mutation; lung cancer; multimodal integration; non-invasive diagnosis; precision medicine; survival prediction.

Graphical abstract

Figure 1

Pipeline of the multimodal LUCID lung cancer diagnostic model (A and B) The model comprises two primary stages. (A) Stage 1 performs binary classification on CT images to identify those with high suspicion of malignancy. (B) Stage 2 integrates four modalities—CT images, textual complaints, lab results, and demographic data—to predict EGFR mutation type or survival time. LUCID is robust to missing modalities and adapts accordingly.

Figure 2

The architecture of the multimodal integration module of LUCID (A) LUCID incorporates four distinct input modalities: CT images, patient’s textual chief complaints, laboratory test results, and demographic information. Each modality is initially encoded into a vector representation before sending into the hierarchical multimodal integration framework. The loss computation encompasses image classification loss, clinical classification loss, and joint loss. (B) Joint-attention mechanism: image and clinical embeddings are merged to create a joint embedding. Multihead attention calculations are then performed to discern the inner-attention across varied features. (C) Cross-attention mechanism: embeddings from one modality serve as the key vector in multihead attention computations, while embeddings from the other modality function as value and query vectors. This facilitates the attention calculations between distinct modalities.

Figure 3

Performance and empirical examples of the stage-1 classification model (A) We evaluated ViT models at varying parameter scales and observed a scaling law in classification capability with the increase of the models’ parameters. (B) A 5-fold validation was conducted during the training of the stage-1 model, and we plotted the AUC along with the 95% CI region. (C) All CT images from a patient (PID:0019770975) diagnosed with stage-4 lung cancer were processed through our stage-1 model. CT images outlined in red indicate the model’s predictions with a high suspicion of lung cancer presence. Data are represented as mean ± 95% CI.

Figure 4

Performance evaluation of LUCID in EGFR type and survival prediction Comparison of LUCID’s accuracy with other methods in two EGFR mutation prediction tasks (N = 1,035) and three survival prediction tasks (N = 952). (A) AUC and method comparisons for distinguishing EGFR-sensitive cases from others (591:444). (B) AUC and method comparisons for distinguishing EGFR wild type from mutant-type cases (599:436), including unimodal and bimodal approaches. (C–E) KM curves and method comparisons for predicting 1-year (835:117), 3-year (469:483), and 5-year (199:753) survival. Data are represented as mean ± 95% CI. p values for KM curves are calculated.

Figure 5

LUCID’s performance on external validation dataset and evaluations of different modalities (A) LUCID demonstrated strong predictive performance with an AUC of 0.876 (95% CI: 0.857–0.893) on the external validation dataset (N = 1,285). (B) Overall performance and three feature exclusion experiments suggest that different feature modalities contribute differently to the model’s predictive capability. Data are represented as mean ± 95% CI.