Robust multi-modal fusion architecture for medical data with knowledge distillation

Abstract

Background: The fusion of multi-modal data has been shown to significantly enhance the performance of deep learning models, particularly on medical data. However, missing modalities are common in medical data due to patient specificity, which poses a substantial challenge to the application of these models.

Objective: This study aimed to develop a novel and efficient multi-modal fusion framework for medical datasets that maintains consistent performance, even in the absence of one or more modalities.

Methods: In this paper, we fused three modalities: chest X-ray radiographs, history of present illness text, and tabular data such as demographics and laboratory tests. A multi-modal fusion module based on pooled bottleneck (PB) attention was proposed in conjunction with knowledge distillation (KD) for enhancing model inference in the case of missing modalities. In addition, we introduced a gradient modulation (GM) method to deal with the unbalanced optimization in multi-modal model training. Finally, we designed comparison and ablation experiments to evaluate the fusion effect, the model robustness to missing modalities, and the contribution of each component (PB, KD, and GM). The evaluation experiments were performed on the MIMIC-IV datasets with the task of predicting in-hospital mortality risk. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC).

Results: The proposed multi-modal fusion framework achieved an AUROC of 0.886 and AUPRC of 0.459, significantly surpassing the performance of baseline models. Even when one or two modalities were missing, our model consistently outperformed the reference models. Ablation of each of the three components resulted in varying degrees of performance degradation, highlighting their distinct contributions to the model's overall effectiveness.

Conclusions: This innovative multi-modal fusion architecture has demonstrated robustness to missing modalities, and has shown excellent performance in fusing three medical modalities for patient outcome prediction. This study provides a novel idea for addressing the challenge of missing modalities and has the potential be scaled to additional modalities.

Keywords: Clinical prediction; Deep learning; Knowledge distillation; Missing modalities; Multi-modal fusion; Transformer.