Pay attention to doctor-patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

Abstract

The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray $+$ 2598 CT $+$ 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray $+$ 494 CT $+$ 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.

Keywords: COVID-19 diagnose; Knowledge attention mechanism; Knowledge embedding; Knowledge-based representation learning.

Fig. 1

Comparison about COVID-19 and non-COVID-19 dialogues.

Fig. 2

An illustrative example of a multi-modal knowledge graph. From left to right, these are the four types of nodes (i.e., X-ray, CT, ultrasound, and text description of diagnose), six meta-paths involved in the given knowledge graph, a multi-modal knowledge graph.

Fig. 3

Architecture of multi-modal knowledge graph model .

Fig. 4

The proposed multi-modal knowledge graph attention embedding model. Given multimodal knowledge graph $\mathcal{G}$, we propose the multimodal attention mechanisms including three parts: ➀ the single-level modality attention and its results denoted as $\{{{\mathbf{f}}^{knowledge}}_{{\Phi }_i},i=0,1,\dots ,C\}$; ➁ the multiple-level modality attention and its embedding denoted as ${{\mathbf{f}}_{multiple}}^{knowledge}$; ➂ cross-level modality attention mechanism that fuse the information of single-level modality and multiple-level modality attentions, and its the embedding matrix denoted as ${\mathbf{f}}^{knowledge}$. Meanwhile, we propose the Temporal Convolutional Self-Attention Network (TCSAN) to handle the inputted multimodal data and get the multimodal sentence vectors ${\mathbf{f}}^{network}$. Then, we get the knowledge-based attention feature vector $\mathbf{f}$. Finally, we use the classifier (in this paper, we use the ResNet-34 [70]) to gain the labels, i.e., ${\hat{y}}_p$.

Fig. 5

The illustration of TCN with dilated causal convolutions.

Fig. 6

The result of parameters experiments.

Fig. 7

Scalability. The training time decreases as the number of training samples increases. Ours takes less training time to converge compared with others.

Fig. 8

The result of robustness analysis. Performance modification using the altered data input over our model .

Fig. 9

The confusion matrix of classifying the asymptomatic infection (COVID-19) cases and non-COVID-19 cases.

Fig. 10

Error analysis.