Dual contrastive learning based image-to-image translation of unstained skin tissue into virtually stained H&E images

Abstract

Staining is a crucial step in histopathology that prepares tissue sections for microscopic examination. Hematoxylin and eosin (H&E) staining, also known as basic or routine staining, is used in 80% of histopathology slides worldwide. To enhance the histopathology workflow, recent research has focused on integrating generative artificial intelligence and deep learning models. These models have the potential to improve staining accuracy, reduce staining time, and minimize the use of hazardous chemicals, making histopathology a safer and more efficient field. In this study, we introduce a novel three-stage, dual contrastive learning-based, image-to-image generative (DCLGAN) model for virtually applying an "H&E stain" to unstained skin tissue images. The proposed model utilizes a unique learning setting comprising two pairs of generators and discriminators. By employing contrastive learning, our model maximizes the mutual information between traditional H&E-stained and virtually stained H&E patches. Our dataset consists of pairs of unstained and H&E-stained images, scanned with a brightfield microscope at 20 × magnification, providing a comprehensive set of training and testing images for evaluating the efficacy of our proposed model. Two metrics, Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), were used to quantitatively evaluate virtual stained slides. Our analysis revealed that the average FID score between virtually stained and H&E-stained images (80.47) was considerably lower than that between unstained and virtually stained slides (342.01), and unstained and H&E stained (320.4) indicating a similarity virtual and H&E stains. Similarly, the mean KID score between H&E stained and virtually stained images (0.022) was significantly lower than the mean KID score between unstained and H&E stained (0.28) or unstained and virtually stained (0.31) images. In addition, a group of experienced dermatopathologists evaluated traditional and virtually stained images and demonstrated an average agreement of 78.8% and 90.2% for paired and single virtual stained image evaluations, respectively. Our study demonstrates that the proposed three-stage dual contrastive learning-based image-to-image generative model is effective in generating virtual stained images, as indicated by quantified parameters and grader evaluations. In addition, our findings suggest that GAN models have the potential to replace traditional H&E staining, which can reduce both time and environmental impact. This study highlights the promise of virtual staining as a viable alternative to traditional staining techniques in histopathology.

Figure 1

Traditional staining workflow (Upper) vs Virtual Staining workflow (Below). The chemical process of staining has been replaced with deep learning-based virtual staining.

Figure 2

Skin tissue dataset. Row 1 shows three whole slide tissue sample pairs (unstained and stained with H&E). Rows 2–5 show patch pairs (unstained and stained with H&E).

Figure 3

Overview of the proposed virtual H&E staining workflow showing preprocessing, training, and inference stages.

Figure 4

DCLGAN architecture involves learning two mappings: G1: A → B and G2: B → A. The encoded half of G1 and G2 is then labeled as G1enc and G2enc, respectively. G1enc and H_A serve as the embedding for A, while G2enc and H_B serve as the embedding for B.

Figure 5

Patchwise contrastive learning maximizes mutual information between input and output patches, enabling one-sided translation in unpaired settings. This is done using a multilayer patchwise contrastive loss that maximizes mutual information between corresponding input and output patches and its corresponding input patch (positive example $v^{+}$) over other random patches (negative example $v^{-}$).

Figure 6

The Patch Noise Contrastive Estimation loss helps the generated image patch to look more like its real input shown in blue) while making it less like the other unrelated patches (shown in red).

Figure 7

The top image demonstrates the patchy appearance of DCLGAN’s output with artifacts. In contrast, the bottom tissue image shows the result after overlapping and blending patches, resulting in a smooth seamless image without any artifacts.

Figure 8

The web application interface for dermatopathologists to evaluate virtual and histologically stained tissues.

Figure 9

Comparative result of different virtual staining methods. The image shows the input unstained tissue, the ground truth H&E stain, and the virtual stains produced by Cycle GAN, CUTGAN, and proposed DCLGAN models.

Figure 10

Unstained, H&E stained and virtually stained patches along with respective quantified average FID and KID scores.

Figure 11

Grading of virtually stained single images for the following seven features: color, resolution, sharpness, contrast, brightness, uniformity of tissue illumination, and absence of artifacts, grading them as Adequate (blue) or Inadequate (pink).

Figure 12

Comparison of assessment made by three graders on the similarity and quality of H&E stained and virtually stained images. The evaluators made a side-by-side comparison between images and rated agreement, disagreement, and neutrality across each pair for different features.