Anatomy-Informed Multimodal Learning for Myocardial Infarction Prediction

Abstract

Goal: In patients with coronary artery disease, the prediction of future cardiac events such as myocardial infarction (MI) remains a major challenge. In this work, we propose a novel anatomy-informed multimodal deep learning framework to predict future MI from clinical data and Invasive Coronary Angiography (ICA) images. Methods: The images are analyzed by Convolutional Neural Networks (CNNs) guided by anatomical information, and the clinical data by an Artificial Neural Network (ANN). Embeddings from both sources are then merged to provide a patient-level prediction. Results: The results of our framework on a clinical study of 445 patients admitted with acute coronary syndromes confirms that multimodal learning increases the predictive power and achieves good performance (AUC: [Formula: see text] & F1-Score: [Formula: see text]), which outperforms the prediction obtained by each modality independently as well as that of interventional cardiologists (AUC: 0.54 & F1-Score: 0.18). Conclusions: To the best of our knowledge, this is the first attempt towards combining multimodal data through a deep learning framework for future MI prediction. Although it demonstrates the superiority of multi-modal approaches over single modality, the results do not yet meet the necessary criteria for practical application.

Keywords: Coronary artery disease; deep learning; invasive coronary angiography; multimodal data; myocardial infarction.

Fig. 1.

Annotated ICA images of a patient; two views for each of three different arteries. The boxes indicate different anatomical segments and the red dots that the segment is responsible of future MI.

Fig. 2.

Anatomy-informed multimodal framework for MI prediction. The patient data is processed by the clinical data block and the two views of each artery are processed by artery blocks. A prediction is provided for each artery and the prediction at the patient level is the maximum of these three predictions. Note that the decision at the artery level is influenced by the clinical data.

Fig. 3.

Annotated ICA images (left) are converted to a raw image (center) and a mask (right) that indicates the different anatomical segments. The mask is created by generating Gaussian functions centered on the sections' rectangle and using the same width and height.

Fig. 4.

Single modality ICA framework for MI prediction. The two views of each artery are processed separately before being concatenated together into a bigger feature map. This new feature map is further processed through convolutional layers and poolings to finally provide a patient-level prediction. The ResConvBlock is the set of convolutional blocks connected with skip connections presented in the section III-A.