Development and evaluation of deep learning-based segmentation of histologic structures in the kidney cortex with multiple histologic stains

Abstract

The application of deep learning for automated segmentation (delineation of boundaries) of histologic primitives (structures) from whole slide images can facilitate the establishment of novel protocols for kidney biopsy assessment. Here, we developed and validated deep learning networks for the segmentation of histologic structures on kidney biopsies and nephrectomies. For development, we examined 125 biopsies for Minimal Change Disease collected across 29 NEPTUNE enrolling centers along with 459 whole slide images stained with Hematoxylin & Eosin (125), Periodic Acid Schiff (125), Silver (102), and Trichrome (107) divided into training, validation and testing sets (ratio 6:1:3). Histologic structures were manually segmented (30048 total annotations) by five nephropathologists. Twenty deep learning models were trained with optimal digital magnification across the structures and stains. Periodic Acid Schiff-stained whole slide images yielded the best concordance between pathologists and deep learning segmentation across all structures (F-scores: 0.93 for glomerular tufts, 0.94 for glomerular tuft plus Bowman's capsule, 0.91 for proximal tubules, 0.93 for distal tubular segments, 0.81 for peritubular capillaries, and 0.85 for arteries and afferent arterioles). Optimal digital magnifications were 5X for glomerular tuft/tuft plus Bowman's capsule, 10X for proximal/distal tubule, arteries and afferent arterioles, and 40X for peritubular capillaries. Silver stained whole slide images yielded the worst deep learning performance. Thus, this largest study to date adapted deep learning for the segmentation of kidney histologic structures across multiple stains and pathology laboratories. All data used for training and testing and a detailed online tutorial will be publicly available.

Keywords: computerized morphologic assessment; deep learning; digital pathology; kidney histologic primitives; large-scale tissue interrogation; renal biopsy interpretation.

Figure 1|. Optimally digitally magnified regions of interest.

The optimal magnification varied for each histologic primitive using patch size of 256 × 256 px: periodic acid–Schiff glomerular unit and tuft, original magnification ×5; proximal and distal tubular segment, original magnification ×10; peritubular capillary, original magnification ×40; and arteries/arterioles, original magnification ×10 (not shown).

Figure 2|. Deep learning (DL) segmentation of glomerular tuft and unit.

DL segmentation for glomerular unit and tuft on whole slide images of formalin-fixed and paraffin-embedded sections from minimal change disease, stained with hematoxylin and eosin (H&E), periodic acid–Schiff (PAS), trichrome (TRI), and silver (SIL). For each stain, the original image overlaid with ground truth is presented on the left, and the DL segmentation is presented on the right. The positive classes are highlighted in bright pink from green transparent mask overlaid on original image. The DL output is specifically tracing the Bowman capsule for glomerular unit and the profile of the capillary wall for the glomerular tuft. The glomerular units and tufts were correctly identified across all types of stains.

Figure 3|. Deep learning (DL) segmentation of proximal and distal tubular segments.

DL segmentation for tubular segments on whole slide images of formalin-fixed and paraffin-embedded sections from minimal change disease, stained with hematoxylin and eosin (H&E), periodic acid–Schiff (PAS), trichrome (TRI), and silver (SIL). For each stain, the original image overlaid with ground truth is presented on the left, and the DL segmentation is presented on the right. The positive classes are highlighted in bright pink from green transparent mask overlaid on original image.

Figure 4|. Deep learning (DL) segmentation of arteries/arterioles and peritubular capillaries.

DL segmentation for arteries/arterioles on whole slide images of formalin-fixed and paraffin-embedded sections from minimal change disease, stained hematoxylin and eosin (H&E), periodic acid–Schiff (PAS), trichrome (TRI), and silver (SIL), and for peritubular capillaries on whole slide images of formalin-fixed and paraffin-embedded sections stained with PAS, with the original image overlaid with ground truth on the left and the DL segmentation on the right. The positive classes are highlighted in bright pink from green transparent mask overlaid on original image.

Figure 5|. Deep learning (DL) Segmentation performance in relation to the morphologic heterogeneity of peritubular capillaries (PTCs).

(a) Most of the peritubular capillaries were small when measured in number of pixels. The size of the peritubular capillaries has an exponential distribution with a long tail from small to large. Each pixel is 0.06 μm2 on tissue, and as observed, most of the PTCs are under 90 μm2. Examples of DL performance on small (c), medium (b), and large (d,e) PCs.

Figure 6|. Deep learning (DL) segmentation of normal histologic primitives on periodic acid–Schiff nephrectomies.

(a) Segmentation of normal glomerular units. (b) Segmentation of proximal (yellow) and distal (green) tubules; rare atrophic tubules were detected by the DL algorithms. (c) Segmentation of arteries/arterioles.

Figure 7|. Segmentation outputs of peritubular capillaries (PTCs) on periodic acid–Schiff (PAS) nephrectomies.

(a) Formalin-fixed and paraffin-embedded sections stained with PAS and CD34 (double stain). (b) Deep learning (DL) segmentation of peritubular capillaries on the same section stained with PAS alone. There is overlap between the CD34 positive stain and the DL detection of peritubular capillaries. Overall, the DL performance was similar to the segmentation accuracy on the testing set for minimal change disease.

Figure 8|. Model performance with increasing number of training annotations.

Number of annotations versus deep learning model performans. The model performance was measured as F-score, dice similarity coefficient (DSC), true positive rate (TPR), predictive positive value (PPV). For histologic primitives such as glomerular tufts, only a small number of annotations was required to construct a robust classifier, in contrast to peritubular capillaries where larger number of annotations were required. The performance metrics for peritubular capillary segmentation increased linearly as more annotations were added. Arteries/arterioles and distal tubules had intermediate rates of convergence with increasing number of annotations.

Figure 9|. Examples of false positive and false negative deep learning (DL) segmentations on periodic acid–Schiff (PAS).

(a) Glomerular unit: DL failed to detect a tangentially cut glomerular unit that does not have a typical round shape (red thick arrow). (b) Artery: section artifact generate a false positive (red thick arrows). (c) Arteries: black arrows show 2 arterioles missed by the pathologist but detected by DL. (d) Arteries: pathologists were instructed to segment artery when lumen was present; however, DL segmentation detected tangentially cut artery (thick black arrow) where only the medium was visible. (e) Peritubular capillaries: a long peritubular capillary reveals only partial DL segmentation at the pixel level. (f) Peritubular capillaries: DL network for peritubular capillaries detects a few glomerular capillaries (false positive; thick red arrow).

Figure 10|. Ground truth annotation for histologic primitives.

Examples of manual annotation on histologic primitives on whole slide images of formalin-fixed and paraffin-embedded sections from minimal change disease, stained with hematoxylin and eosin (H&E), periodic acid–Schiff (PAS), trichrome (TRI), and silver (SIL), and corresponding binary masks (black and white pictures) are shown.

Figure 11|. Flowchart of the workflow of deep learning (DL) experimental pipeline for each stain and use case.

(a) Whole slide images (WSIs) were selected for generation of training, validation, and testing data. (b) Regions of interest were cropped from original WSIs with 40× digital magnification. (c) Ground truth labels were generated by pathologists for training, and overlapping patches of size 256 × 256 px (0.24 μm/px) containing both image data and ground truth annotation information were cropped from the training and validation images (as shown in black boxes). (d) For each path, a randomized data augmentation method is introduced to account for (i) size variation of primitives, (ii) stain variations, and (iii) tissue variations (e.g. thickness). (e) All the training patches were passed to U-Net on PyTorch for training, and validation patches were used to generate loss and accuracy measures for each epoch trained to evaluate model performance. Finally, the epoch that yielded the lowest loss on the validation data was selected for generation of test results.