Pathology Image Analysis Using Segmentation Deep Learning Algorithms

Abstract

With the rapid development of image scanning techniques and visualization software, whole slide imaging (WSI) is becoming a routine diagnostic method. Accelerating clinical diagnosis from pathology images and automating image analysis efficiently and accurately remain significant challenges. Recently, deep learning algorithms have shown great promise in pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many machine learning algorithms, including convolutional neural networks, have been proposed to automatically segment pathology images. Among these algorithms, segmentation deep learning algorithms such as fully convolutional networks stand out for their accuracy, computational efficiency, and generalizability. Thus, deep learning-based pathology image segmentation has become an important tool in WSI analysis. In this review, the pathology image segmentation process using deep learning algorithms is described in detail. The goals are to provide quick guidance for implementing deep learning into pathology image analysis and to provide some potential ways of further improving segmentation performance. Although there have been previous reviews on using machine learning methods in digital pathology image analysis, this is the first in-depth review of the applications of deep learning algorithms for segmentation in WSI analysis.

Figure 1

Illustration of convolution operation in a typical segmentation neural network. A: The structure of an example segmentation neural network. The input is a 512 × 512 × 3 hematoxylin and eosin–stained lung cancer pathology image from the National Lung Screening Trial (NLST). After the first convolution operation with n1 kernels, a 256 × 256 × n1 feature map was generated (the height and width differences between input and feature map 1 are due to stride in the convolution operation, which means the step size of the sliding window is equal to the stride value. In this case, stride = 2. Feature map 1 was then used as input for the second convolution operation with n2 kernels, which yields a 128 × 128 × n2 feature map. In this example, the bounding box, category, and mask of each object is then determined based on feature map 2. Mapping all of the identified objects back to their positions in the input image generates the final segmentation output. In this case, the objects to be identified are cell nuclei, and categories are different cell types. Green: tumor nuclei; blue: lymphocyte nuclei; red: stroma nuclei. B: The first step of the convolution operation shown in A. For illustration purposes, only two of the n1 kernels and corresponding outputs in the first convolutional (Conv) layer are plotted. Each kernel is used to generate one layer (256 × 256 × 1) of feature map 1. The values of the kernels (middle) are retrieved from a convolutional neural network model trained with pathology images. There are 7 × 7 × 3 (width × height × R/G/B channel) = 147 values in each kernel. A receptive field (left) is the part of the image covered by the kernel, and it has the same size (147 values) as the kernel. An output pixel is calculated by summing the products of the pixel-wise multiplications between the values in one receptive field and the corresponding values in the kernel (ie, summation of 147 products). In this illustration, the pixel indicated by the brown arrow in the top panel is calculated from the receptive field surrounded by the yellow rectangle and kernel 1, whereas the pixel indicated by the brown arrow in the bottom panel is calculated from the same receptive field and kernel 2. By sliding the receptive fields along the width and height dimensions, the 2-dimensional output (also called the feature map) can be obtained. In this example, the output of kernel 1 reveals fibrous-like structures whereas the output of kernel n1 reveals nuclei-like structures.

Figure 2

Flow chart of pathology image analysis using segmentation deep learning algorithms. The example pathology image is a hematoxylin and eosin–stained image from the National Lung Screening Trial (NLST). Green arrows are steps that are necessary for both training and application phases; orange arrows are steps that are only performed during the neural network training phase; and blue arrows are steps only performed during the application process. WSI, whole slide image.

Figure 3

Example of nuclei segmentation in a pathology image. Input is a hematoxylin and eosin–stained image patch (left panel) from the National Lung Screening Trial (NLST). Output is either semantic (middle panel) or instance (right panel) segmentation result. For semantic segmentation, each pixel is assigned a class, whereas for instance segmentation, only objects (nuclei) are picked out and each object is assigned a class.

Figure 4

Illustration of encoder and decoder network for semantic segmentation (A) and instance segmentation (B). Specifically, Mask R-CNN structure is used as an example of instance segmentation. ROI, region of interest; ROIAlign, a specific network layer that can efficiently compute feature maps within the candidate regions of interest; RPN, region proposal network, a network to output possible regions (proposal) for objects of interest.