Tackling stain variability using CycleGAN-based stain augmentation

Abstract

Background: Considerable inter- and intra-laboratory stain variability exists in pathology, representing a challenge in development and application of deep learning (DL) approaches. Since tackling all sources of stain variability with manual annotation is not feasible, we here investigated and compared unsupervised DL approaches to reduce the consequences of stain variability in kidney pathology.

Methods: We aimed to improve the applicability of a pretrained DL segmentation model to 3 external multi-centric cohorts with large stain variability. In contrast to the traditional approach of training generative adversarial networks (GAN) for stain normalization, we here propose to tackle stain variability by data augmentation. We augment the training data of the pretrained model by the stain variability using CycleGANs and then retrain the model on the stain-augmented dataset. We compared the performance of i/ the unmodified pretrained segmentation model with ii/ CycleGAN-based stain normalization, iii/ a feature-preserving modification to ii/ for improved normalization, and iv/ the proposed stain-augmented model.

Results: The proposed stain-augmented model showed highest mean segmentation accuracy in all external cohorts and maintained comparable performance on the training cohort. However, the increase in performance was only marginal compared to the pretrained model. CycleGAN-based stain normalization suffered from encoded imperceptible information into the normalizations that confused the pretrained model and thus resulted in slightly worse performance.

Conclusions: Our findings suggest that stain variability can be tackled more effectively by augmenting data by it than by following the commonly used approach of normalizing the stain. However, the applicability of this approach providing only a rather slight performance increase has to be weighted against an additional carbon footprint.

Keywords: Deep learning; Digital pathology; Kidney; Segmentation; Stain augmentation; Stain normalization.

Fig. 1

Overview of stain variations in all cohorts.

Fig. 2

Approaches for improved generalization of the pretrained segmentation model to external cohorts. To improve the pretrained model S (i/), CycleGANs are trained for translation between its training cohort AC and the external cohorts, respectively. Stain normalization then uses the CycleGANs to translate the external domains into the training cohort AC for improved application (ii/ + iii/). In contrast, the proposed stain augmentation augments the annotated training data of S by the external stain variations using the CycleGANs (iv/). Then, a new and cohort-independent segmentation model is trained on the stain-augmented annotated training data.

Fig. 3

Qualitative segmentation results in all external cohorts. For each external, multi-centric cohort, images with 2 representative stain variations are selected (column 1). Their predictions by the proposed stain-augmented model are shown (column 2). They are colored in accordance with Fig. 1, however tubules are colored randomly here to assess the feasibility of their instance separation.

Fig. 4

Qualitative normalization and segmentation results of all approaches. The pretrained segmentation model is denoted by “Baseline”, the proposed stain augmentation by “StainAugm”, CycleGAN-based stain normalization by “StainNorm”, and its feature-preserving modification by “w/ SegNet”. The stain-normalized translations by the latter 2 approaches are denoted by “Norm_StainNorm” and “Norm_{w/ SegNet}”, respectively. Interfering information got encoded into the stain-normalized images that confused the pretrained segmentation model and led to erroneous predictions (a few marked by red arrows).