Content-Based Histopathological Image Retrieval

Abstract

Feature descriptors in histopathological images are an important challenge for the implementation of Content-Based Image Retrieval (CBIR) systems, an essential tool to support pathologists. Deep learning models like Convolutional Neural Networks and Vision Transformers improve the extraction of these feature descriptors. These models typically generate embeddings by leveraging deeper single-scale linear layers or advanced pooling layers. However, these embeddings, by focusing on local spatial details at a single scale, miss out on the richer spatial context from earlier layers. This gap suggests the development of methods that incorporate multi-scale information to enhance the depth and utility of feature descriptors in histopathological image analysis. In this work, we propose the Local-Global Feature Fusion Embedding Model. This proposal is composed of three elements: (1) a pre-trained backbone for feature extraction from multi-scales, (2) a neck branch for local-global feature fusion, and (3) a Generalized Mean (GeM)-based pooling head for feature descriptors. Based on our experiments, the model's neck and head were trained on ImageNet-1k and PanNuke datasets employing the Sub-center ArcFace loss and compared with the state-of-the-art Kimia Path24C dataset for histopathological image retrieval, achieving a Recall@1 of 99.40% for test patches.

Keywords: content-based image retrieval; feature embedding; feature fusion; histopathological image; transfer learning.

Figure 1

The LGFFEM architecture comprises a pre-trained backbone as a feature extractor from multi-scale stages, a trainable neck consisting of layers for local–global feature fusion, and a pooling head composed of trainable GeM mini-heads for each multi-scale fused feature from the neck.

Figure 2

Illustration of the bottleneck operation for the Local and Global Aggregators and the pooling GeM mini-head. (a) Detailed schematic of the Global Feature Aggregator Unit. (b) Detailed schematic of the Local Feature Aggregator Unit. (c) Detailed schematic of the mini-head unit from the GeM head.

Figure 3

Query image selected for the class set S0 and their first retrieve image from the Kimia Patch24C dataset. (a) Query image ID S0-1. (b) First image retrieved ID S0-2.

Figure 4

Grad-CAM applied to the first layer of the neck used in strategy C for the first image’s retrieved ID S0-2. (a) Grad-CAM applied to the outer aggregation fusion node $P_{1\_2}$ in Layer 1. (b) Grad-CAM applied to the outer aggregation fusion node $P_{2\_2}$ in Layer 1. (c) Grad-CAM applied to the outer aggregation fusion node $P_{3\_2}$ in Layer 1. (d) Grad-CAM applied to the outer aggregation fusion node $P_{4\_2}$ in Layer 1. (e) Grad-CAM applied to the collapse of all outer aggregation fusion nodes in Layer 1.

Figure 5

Grad-CAM applied to the second layer of the neck used in strategy C for the first image’s retrieved ID S0-2. (a) Grad-CAM applied to the outer aggregation fusion node $P_{1\_2}$ in Layer 2. (b) Grad-CAM applied to the outer aggregation fusion node $P_{2\_2}$ in Layer 2. (c) Grad-CAM applied to the outer aggregation fusion node $P_{3\_2}$ in Layer 2. (d) Grad-CAM applied to the outer aggregation fusion node $P_{4\_2}$ in Layer 2. (e) Grad-CAM applied to the collapse of all outer aggregation fusion nodes in Layer 2.

Figure 6

Grad-CAM applied to the third layer of the neck used in strategy C for the first image’s retrieved ID S0-2. (a) Grad-CAM applied to the outer aggregation fusion node $P_{1\_2}$ in Layer 3. (b) Grad-CAM applied to the outer aggregation fusion node $P_{2\_2}$ in Layer 3. (c) Grad-CAM applied to the outer aggregation fusion node $P_{3\_2}$ in Layer 3. (d) Grad-CAM applied to the outer aggregation fusion node $P_{4\_2}$ in Layer 3. (e) Grad-CAM applied to the collapse of all outer aggregation fusion nodes in Layer 3.

Figure 7

Visualization of the 2D projection of embeddings from the Kimia Patch24C test dataset using strategy C. Each dot represents an image, and each color represents a class of tissue.