WWFedCBMIR: World-Wide Federated Content-Based Medical Image Retrieval

Abstract

The paper proposes a federated content-based medical image retrieval (FedCBMIR) tool that utilizes federated learning (FL) to address the challenges of acquiring a diverse medical data set for training CBMIR models. CBMIR is a tool to find the most similar cases in the data set to assist pathologists. Training such a tool necessitates a pool of whole-slide images (WSIs) to train the feature extractor (FE) to extract an optimal embedding vector. The strict regulations surrounding data sharing in hospitals makes it difficult to collect a rich data set. FedCBMIR distributes an unsupervised FE to collaborative centers for training without sharing the data set, resulting in shorter training times and higher performance. FedCBMIR was evaluated by mimicking two experiments, including two clients with two different breast cancer data sets, namely BreaKHis and Camelyon17 (CAM17), and four clients with the BreaKHis data set at four different magnifications. FedCBMIR increases the F1 score (F1S) of each client from 96% to 98.1% in CAM17 and from 95% to 98.4% in BreaKHis, with 11.44 fewer hours in training time. FedCBMIR provides 98%, 96%, 94%, and 97% F1S in the BreaKHis experiment with a generalized model and accomplishes this in 25.53 fewer hours of training.

Keywords: breast cancer; computer-aided diagnosis; content-based medical image retrieval (CBMIR); convolutional auto-encoder (CAE); digital pathology; federated learning (FL); histopathological images; whole-slide images (WSIs).

Figure 1

An overview of the use case of a worldwide CBMIR. Pathologists send their query (Q) to the worldwide CBMIR since they need a second opinion to make a more confident decision. Then, the model retrieved top K similar images (S-R), and the pathologists can obtain a second opinion from whole over the world.

Figure 2

A comprehensive illustration of the entire process in a CBMIR, demonstrating the utilization of DL models to acquire images from a hospital and offer a second opinion for pathologists.

Figure 3

The structure of the custom-built CAE. The stride in the encoder = $[1,2,2,2]$, in the bottleneck = $[1,1,1,1]$, in the decoder related to the encoder = $[2,2,2,1]$. The kernel size of the layers in all parts of the structure and for each layer is 3.

Figure 4

The pipeline of CBMIR. It contains three important sections, namely (1) FE, (2) indexing and saving, and (3) similarity measure and search.

Figure 5

The FedCBMIR pipeline consists of four main steps. Step 1: the server initializes weights, and then sends to client for local training. Step 2: client starts local training. Step 3: client updates local weights to the server side. Step 4: the server side aggregates and updates the distributed weights. (a) An overview of the FedCBMIR pipeline with two clients training, fed with BreaKHis 40× and CAM17 data sets. (b) An overview of the FedCBMIR pipeline with four clients training over clusters at universities and companies with BreaKHis in four different magnifications.

Figure 6

Three random queries from Hospital 5 of CAM17 (test set). Corresponding to each query, the top 5 images are shown from four other hospitals with the most similar patterns to the query. The green and red lines around the retrieved images explain the correct and wrong retrieved images.

Figure 7

(a) shows the results of local training on CAM17 in the TY server. (b) is the result of the searching task in CAM17 by applying the well-trained FedCBMIR model from the first experiment.

Figure 8

(a–d) show the CMs as a result of local training and searching at the same magnification. (e–h) are the CMs of FL models. The reported results are with top K retrieved images. In all CMs, "0” and “1” indicate “Benign” and “Malignant”, respectively. “True labels” and “Predicted labels” correspond to the query and the retrieved labels, accordingly.

Figure 9

BreaKHis images at four different magnification levels (40×, 100×, 200×, and 400×). The higher magnification offers increased access to relevant information with a reduced field of view.

Figure 10

An indirect comparison between the results of FedCBMIR in both experiments and some recent methods for different amounts of K.

Figure 11

Five lines of random histopathological WSIs with their magnifications. The first column is the query, and the following five columns show the retrieved images. This figure brings a proper overview of Sen2. The retrieved image with the same and different labels as the query is indicated by the green and red borders, accordingly.