Nuclear IHC enumeration: A digital phantom to evaluate the performance of automated algorithms in digital pathology

Abstract

Automatic and accurate detection of positive and negative nuclei from images of immunostained tissue biopsies is critical to the success of digital pathology. The evaluation of most nuclei detection algorithms relies on manually generated ground truth prepared by pathologists, which is unfortunately time-consuming and suffers from inter-pathologist variability. In this work, we developed a digital immunohistochemistry (IHC) phantom that can be used for evaluating computer algorithms for enumeration of IHC positive cells. Our phantom development consists of two main steps, 1) extraction of the individual as well as nuclei clumps of both positive and negative nuclei from real WSI images, and 2) systematic placement of the extracted nuclei clumps on an image canvas. The resulting images are visually similar to the original tissue images. We created a set of 42 images with different concentrations of positive and negative nuclei. These images were evaluated by four board certified pathologists in the task of estimating the ratio of positive to total number of nuclei. The resulting concordance correlation coefficients (CCC) between the pathologist and the true ratio range from 0.86 to 0.95 (point estimates). The same ratio was also computed by an automated computer algorithm, which yielded a CCC value of 0.99. Reading the phantom data with known ground truth, the human readers show substantial variability and lower average performance than the computer algorithm in terms of CCC. This shows the limitation of using a human reader panel to establish a reference standard for the evaluation of computer algorithms, thereby highlighting the usefulness of the phantom developed in this work. Using our phantom images, we further developed a function that can approximate the true ratio from the area of the positive and negative nuclei, hence avoiding the need to detect individual nuclei. The predicted ratios of 10 held-out images using the function (trained on 32 images) are within ±2.68% of the true ratio. Moreover, we also report the evaluation of a computerized image analysis method on the synthetic tissue dataset.

Fig 1. Examples of positive and negative objects used in creation of synthetic images.

All objects were cropped from Follicular Lymphoma whole slide images stained for Ki67.

Fig 2. Examples of synthetic images generated by the proposed method.

Left) An example image with only a few Ki67 positive nuclei. Right) An example image with higher concentration of Ki67 positive nuclei.

Fig 3. Bland Altman Plots for r_n.

Bias and variability of the pathologists’ estimates of r_n increased with the percentage of positive nuclei. However, the bias and variability of the computer’s estimate was relatively smaller than the pathologists’ estimate of r_n.

Fig 4. Bland Altman Plots for r_a.

The absolute bias and the variability of the pathologists’ estimates of r_a decreased with the increasing percentage of positive nuclei. This shows that for images with higher concentrations of positive nuclei, pathologists’ estimates deviate from accuracy in ways that are not present in the algorithm’s estimates.

Fig 5. Bland Altman Plots for r_t.

There was a strong increasing linear relationship between pathologist bias in estimating r_t and the percentage of positive nuclei. In contrast, there was an almost perfect inverse linear relationship between the bias of the computer’s estimates and percentage of positive nuclei.

Fig 6. The plot shows a function which facilitates the mapping of r_a → r_n.

The horizontal axis corresponds to different values of r_a while the vertical axis represents D, i.e. error. Each individual dot represents the error between r_a and r_n for the training images. The solid line corresponds to the mapping function Ψ which facilitates the mapping of r_a to r_n while the dotted lines represent the confidence interval.