Deep learning-based transformation of H&E stained tissues into special stains

Abstract

Pathology is practiced by visual inspection of histochemically stained tissue slides. While the hematoxylin and eosin (H&E) stain is most commonly used, special stains can provide additional contrast to different tissue components. Here, we demonstrate the utility of supervised learning-based computational stain transformation from H&E to special stains (Masson's Trichrome, periodic acid-Schiff and Jones silver stain) using kidney needle core biopsy tissue sections. Based on the evaluation by three renal pathologists, followed by adjudication by a fourth pathologist, we show that the generation of virtual special stains from existing H&E images improves the diagnosis of several non-neoplastic kidney diseases, sampled from 58 unique subjects (P = 0.0095). A second study found that the quality of the computationally generated special stains was statistically equivalent to those which were histochemically stained. This stain-to-stain transformation framework can improve preliminary diagnoses when additional special stains are needed, also providing significant savings in time and cost.

Fig. 1. Overview of deep learning-based H&E stain transformation into special stains.

Histochemical staining of H&E is digitally transformed using a deep neural network into the special stains: (i) generation of JMS (purple arrow); (ii) generation of MT (red arrow); (iii) generation of PAS (blue arrow).

Fig. 2. Deep neural networks used to generate the training data for the stain transformation network.

a Virtual staining network (pink arrow) which can generate both the H&E and special stain images. b Style transfer network (green arrow) that is used just to augment the training data. c Scheme used to train the stain transformation network. During its training, the stain transformation network is randomly given, as the input, either the virtually stained H&E tissue, or an image of the same field of view after passing through one of the eight style transfer networks. A perfectly matched virtually stained tissue image with the desired special stain (in this example shown: PAS) is used as the ground truth to train this neural network.

Fig. 3. Overview of the study design.

Phase 1 shows the initial portion of the study where three pathologists review H&E WSIs of N = 58 different tissue sections (each from a unique patient). After a >3-week washout period, the second phase of diagnosis is performed, where the same three pathologists view the same WSIs, where, in addition to the H&E, the special stains generated by the stain transformation technique (PAS, Masson’s Trichrome, Jones) are provided as well. After an additional >3-week washout period, the third phase of diagnosis is performed, where the same three pathologists again review the same WSIs. For this phase, instead of using special stains generated through the stain transformation technique, the images of all four stains (H&E, PAS, Masson’s Trichrome, and Jones) come from histochemically stained serial sections. (i) Generation of JMS. (ii) Generation of MT. (iii) Generation of PAS.

Fig. 4. Visualization of the improvements, concordances, and discordances by case number for the two comparisons.

a Comparison of H&E only vs. H&E and the three stain-transformed special stains coming from the same tissue section. The use of the three stain-transformed special stains results in a statistically significant improvement over H&E only (P = 0.0095). b Comparison of H&E only vs. H&E and the three special stains (all histochemically stained) coming from serial tissue sections. The use of the three histochemically stained special stains results in a statistically significant improvement over H&E only (P = 0.0003). P values were calculated using a one-sided t-test. No adjustments for multiple comparisons were needed.

Fig. 5. Examples of improved diagnoses fostered by the stain-transformed special stains.

We report here WSIs that are generated using the stain transformation technique. In this case, the addition of the computationally generated special stains improved all three of the diagnoses made by the pathologists. The red arrows point to a region, where the special stains help highlight inflammatory cells within the tubule, otherwise, the boundary of the tubules cannot be seen with the H&E stain only. (i) Generation of JMS. (ii) Generation of MT. (iii) Generation of PAS. A total of 58 cases were viewed by three pathologists to perform the statistical analysis.

Fig. 6. Examples of improved and discordant diagnosis achieved by the stain-transformed special stains.

a Example of improved diagnosis fostered by the stain-transformed special stains. For case #2 (in Fig. 4 and the Supplementary Data 1), the basement membrane changes that are characteristic of membranous nephropathy (subepithelial spikes and basement membrane holes) are only appreciated after reviewing the stain-transformed JMS. The bottom images exemplify histochemically stained images of adjacent serial sections of the patient sample; that is why they correspond to different sections within the tissue block. b Example of the discordance demonstrated between the H&E and computationally generated special stains for case #1 (in Fig. 4 and the Supplementary Data 1). In this field of view, the fibrin thrombi are gray-yellow in color on the stain-transformed PAS stain rather than pink-red. (i) Generation of JMS. (ii) Generation of MT. (iii) Generation of PAS. A total of 58 cases were viewed by three pathologists to perform the statistical analysis.