Deep learning for necrosis detection using canine perivascular wall tumour whole slide images

Abstract

Necrosis seen in histopathology Whole Slide Images is a major criterion that contributes towards scoring tumour grade which then determines treatment options. However conventional manual assessment suffers from inter-operator reproducibility impacting grading precision. To address this, automatic necrosis detection using AI may be used to assess necrosis for final scoring that contributes towards the final clinical grade. Using deep learning AI, we describe a novel approach for automating necrosis detection in Whole Slide Images, tested on a canine Soft Tissue Sarcoma (cSTS) data set consisting of canine Perivascular Wall Tumours (cPWTs). A patch-based deep learning approach was developed where different variations of training a DenseNet-161 Convolutional Neural Network architecture were investigated as well as a stacking ensemble. An optimised DenseNet-161 with post-processing produced a hold-out test F1-score of 0.708 demonstrating state-of-the-art performance. This represents a novel first-time automated necrosis detection method in the cSTS domain as well specifically in detecting necrosis in cPWTs demonstrating a significant step forward in reproducible and reliable necrosis assessment for improving the precision of tumour grading.

Figure 1

In (a), Annotations by ”Annotator 1” and ”Annotator 2” applied to the same canine Perivascular Wall Tumour (cPWT) Whole Slide Image (WSI). For the patch extraction process, binary masks (or maps) are generated, (shown in (b). A necrosis mask is created, highlighting the intersection agreement between both annotators, when considering a region as necrotic. Any disagreement is dismissed from the necrosis and negative binary masks. From applying Otsu thresholding we dismissed any non-tissue related regions and by using the intersection agreement for both annotators, we created a ”negative mask”, highlighting in white regions that do not contain necrosis. We used these masks to extract patches, as shown in (c). In this case we extract 10x magnification necrosis and negative patches of 256 $\times$ 256 pixels.

Figure 2

(a) Bottleneck feature extraction using DenseNet-161. A patch size of 256 x 256 pixels is fed into a DenseNet-161 feature extractor, where bottleneck features are obtained. These features are then fed into a classification layer for further training and validation, classifying necrosis or negative patches. (b) Hard negative mining approach to train the model with additional ”difficult” examples presented to the network. (c) Stacking ensemble. The input X is fed into M base-level member models: DenseNet-161 model, the DenseNet-161 model with augmentations and the hard negative mining model. The prediction outputs of these models ${\widehat{y}}_M$ are combined and fed into a logistic regression meta-model as new feature inputs. New coefficients are learnt in this logistic regression model, before final predictions are output ${\widehat{y}}^{\mathit{final}}$.

Figure 3

Histograms of the initial classification results based on a standard 0.5 (50%) probability decision threshold. Depicted are true negatives (TN), false positives (FP), false negatives (FN) and true positives (TP) for the DenseNet-161 model. On the left side depicts histogram plots of TN and FP for each validation fold, whereas on the right side depicts histogram plots of FN and TP. These combinations were chosen for the plots as they complement each other. It can be seen that all three folds are characteristically similar in distribution. TN and TP predictions typically produce high probabilities, as can be seen by the frequency of such predicted probabilities. Increasing the probability threshold would increase the number of true negatives and reduce the number of false positives. However, this would subsequently increase the number of false negatives and reduce the number of true positives.

Figure 4

Line graphs that depict the sensitivity, specificity and weighted F1-score calculated for each probability threshold, for the three validation folds from the DenseNet-161 and ensemble models. To determine the optimal probability threshold, we choose the threshold with the highest F1-score. In the above plots, these are denoted as ”Best threshold”. For example, for the ensemble model, in validation fold 1, this threshold was 0.86, for fold 2 it was 0.65 and for fold 3 it was 0.97.

Figure 5

The post-processing step to remove predicted single necrosis tiles is depicted. The necrosis predictions are applied to a binary mask which is a downsized binary map of the original WSI, by a factor of 32x, for computational efficiency. Connected components analysis is subsequently performed, where if a tile of a fixed size (in this case an area of 32 pixels squared) is not connected to other tiles, horizontally, vertically, or diagonally, it is removed from the mask. Final predictions are updated based on using these binary masks.

Figure 6

Sample Whole Slide Image (WSI) spatial confusion maps before and after applying optimal threshold (determined from the fold 3 validation set) and post-processing; removing single tile predictions. The left side images shows predictions from the DenseNet-161 and ensemble models with the standard 50% probability decision thresholds. The right side shows predictions after applying the optimal threshold and post-processing. True positives (TP) are displayed in red, false negatives (FN) in green, false positives (FP) in yellow and true negatives (TN) in clear.