Hybrid DAER Based Cross-Modal Retrieval Exploiting Deep Representation Learning

Abstract

Information retrieval across multiple modes has attracted much attention from academics and practitioners. One key challenge of cross-modal retrieval is to eliminate the heterogeneous gap between different patterns. Most of the existing methods tend to jointly construct a common subspace. However, very little attention has been given to the study of the importance of different fine-grained regions of various modalities. This lack of consideration significantly influences the utilization of the extracted information of multiple modalities. Therefore, this study proposes a novel text-image cross-modal retrieval approach that constructs a dual attention network and an enhanced relation network (DAER). More specifically, the dual attention network tends to precisely extract fine-grained weight information from text and images, while the enhanced relation network is used to expand the differences between different categories of data in order to improve the computational accuracy of similarity. The comprehensive experimental results on three widely-used major datasets (i.e., Wikipedia, Pascal Sentence, and XMediaNet) show that our proposed approach is effective and superior to existing cross-modal retrieval methods.

Keywords: cross-modal retrieval; data augmentation; dual attention network; enhanced relation network.

Figure 1

The framework of the DAER approach for cross-modal retrieval.

Figure 2

The structure of bottleneck module with dual spatial attention network.

Figure 3

Normal distribution with two different powers (left: power = 1; right: power = 0.6).

Figure 4

Example of retrieval tasks using the proposed DAER.

Figure 5

Comparison of DAER with the selected methods in three datasets.

Figure 6

Comparison of mAP values of each method used in two tasks in three datasets.

Figure 7

The improvement of our proposed approach in three databases.