A multi-modal fusion model with enhanced feature representation for chronic kidney disease progression prediction

Abstract

Artificial intelligence (AI)-based multi-modal fusion algorithms are pivotal in emulating clinical practice by integrating data from diverse sources. However, most of the existing multi-modal models focus on designing new modal fusion methods, ignoring critical role of feature representation. Enhancing feature representativeness can address the noise caused by modal heterogeneity at the source, enabling high performance even with small datasets and simple architectures. Here, we introduce DeepOmix-FLEX (Fusion with Learning Enhanced feature representation for X-modal or FLEX in short), a multi-modal fusion model that integrates clinical data, proteomic data, metabolomic data, and pathology images across different scales and modalities, with a focus on advanced feature learning and representation. FLEX contains a Feature Encoding Trainer structure that can train feature encoding, thus achieving fusion of inter-feature and inter-modal. FLEX achieves a mean AUC of 0.887 for prediction of chronic kidney disease progression on an internal dataset, exceeding the mean AUC of 0.727 using conventional clinical variables. Following external validation and interpretability analyses, our model demonstrated favorable generalizability and validity, as well as the ability to exploit markers. In summary, FLEX highlights the potential of AI algorithms to integrate multi-modal data and optimize the allocation of healthcare resources through accurate prediction.

Keywords: chronic kidney disease; computational pathology; deep learning; multi-modal; multi-omics; progression prediction.

Figure 1

Overview of the FLEX. (a) The multimodal data are derived from EMR, urine samples and kidney biopsy samples that are retained by patients during medical procedures. Patients diagnosed with chronic kidney disease will receive regular follow-up to document disease progression over a three-year period. (b) Four modalities of data were extracted from records and samples. Clinical data from EMR, proteomics and metabolomics data from urine samples, and PASM-stained digital pathology images from kidney samples, respectively. (c) The FLEX receives the preprocessed data of the four modalities, uses an end-to-end training approach to extract the representative vectors for four modalities separately, and fuses the vectors to generate a multimodal representation to be fed into a classifier stacked with two fully-connected layers to predict the three-year progression of a patient’s CKD. (d) FLEX can perform the downstream tasks of progression prediction, modal importance analysis, feature importance interpretation, and image visualization. (C: Clinical data, P: Proteomics data, M: Metabolomics data, I: Pathology images).

Figure 2

Performance of the FLEX on internal dataset. (a) Receiver operating characteristic (ROC) plots for FLEX with five-fold cross-validation based on four-modal data for predicting three-year disease progression in CKD patients. (b) AUC performance distribution of different modal combinations in five-fold cross-validation for FLEX. (c) Comparison of KFRE, FLEX-C (FLEX with only clinical modality inputs) and FLEX performance using mean AUC. (d) Performance comparison of FLEX using different modal fusion methods. (e) Performance comparison of FLEX using different base image models. (f) Comparison of mean AUC performance between missing individual modal data and replacing the missing modality with meta module for both cases. (g) Performance comparisons and percentage statistics using gender, if IgAN, age and eGFR to categorize patients into different groups.

Figure 3

Interpretation of the output of the FLEX with progression group. (a) The modal importance of progression group (n = 106) is in descending order of metabolomics, clinical, proteomics, and images data. (b) Top 10 features in clinical data. (c) Top 10 features in proteomics data. (d) Top 10 features in metabolomics data (^* used to distinguish different metabolites with the same chemical formula). (e) Pathology images of whole slide images (WSI), heat map and regions of interest (ROI). The progression group was more focus on renal tubular atrophy, interstitial fibrosis, and white vacuoles in the tubules.

Figure 4

Interpretation of the output of the FLEX with non-progression group. (a) The modal importance of non-progression group (n = 153) is in descending order of proteomics, clinical, metabolomics, and images data. (b) Top 10 features in clinical data. (c) Top 10 features in proteomics data. (d) Top 10 features in metabolomics data. (^* used to distinguish different metabolites with the same chemical formula.) (e) Pathology images of whole slide images (WSI), heat map and regions of interest (ROI). The non-progression group was more focus on normal tissue areas.

Figure 5

Validation of FLEX on internal and external datasets. (a) The modal importance of all patients is in descending order of clinical, metabolomics, proteomics, and images data on internal dataset. (b) Top 10 features in clinical data on internal dataset. (c) Top 10 features in proteomics data on internal dataset. (d) Top 10 features in metabolomics data on internal dataset. (e) Comparison of AUCs for tests in the internal and external datasets based on data from the three modalities. (f) Top 10 features in clinical data on external dataset. (g) Top 10 features in proteomics data on external dataset. (h) Top 10 features in metabolomics data on external dataset (^* used to distinguish different metabolites with the same chemical formula).