Optimized generation of high-resolution phantom images using cGAN: Application to quantification of Ki67 breast cancer images

Abstract

In pathology, Immunohistochemical staining (IHC) of tissue sections is regularly used to diagnose and grade malignant tumors. Typically, IHC stain interpretation is rendered by a trained pathologist using a manual method, which consists of counting each positively- and negatively-stained cell under a microscope. The manual enumeration suffers from poor reproducibility even in the hands of expert pathologists. To facilitate this process, we propose a novel method to create artificial datasets with the known ground truth which allows us to analyze the recall, precision, accuracy, and intra- and inter-observer variability in a systematic manner, enabling us to compare different computer analysis approaches. Our method employs a conditional Generative Adversarial Network that uses a database of Ki67 stained tissues of breast cancer patients to generate synthetic digital slides. Our experiments show that synthetic images are indistinguishable from real images. Six readers (three pathologists and three image analysts) tried to differentiate 15 real from 15 synthetic images and the probability that the average reader would be able to correctly classify an image as synthetic or real more than 50% of the time was only 44.7%.

Fig 1. Training of discriminator network.

For real examples, we used the real images and their segmentation/annotation masks (Mⁱ, $I_r^i$) as an input. The green and red colored annotations correspond to Ki67 positive and Ki67 negative nuclei, respectively. For fake examples, we applied a two-step procedure. In Step 1, we used generator (U-net) algorithm to create a synthetic image by using the segmentation/annotation. In Step 2, the output of the generator and initial segmentation (Mⁱ, $I_g^i$) are used as an input for D.

Fig 2. Used neural network framework for generator, G.

Fig 3

Fully synthetic images (e-g). We created several toy data to generate synthetic images with different characteristics by using annotation based input (a) and segmentation based input (b and c).

Fig 4. Example (a) real image, (b) segmentation result based on [5], (c) synthetic image used for evaluation of computerized quantitative method, (d) visual ImmunoRatio output for the real image, visual ImmunoRatio output for synthetic image.

Fig 5

Example images (a) original image used for annotation (b) a dot based annotation, (c) cGAN generated synthetic image from (b). (d) Original image used for segmentation (e) segmentation result using [23], (f) cGAN generated image from (e).

Fig 6. Bland-Altman plot comparing ImmunoRatio values of real and artificial images.

Shaded region corresponds to the limits of agreement.

Fig 7. An example case where the immunoRatio values are different in the real and synthetic images.

The upper left nucleus in the real image (a) was missed by the segmentation result (b) based on [5]. Therefore the synthetic image (c) was not including that nucleus.