Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer

Abstract

Advances in the quality of whole-slide images have set the stage for the clinical use of digital images in anatomic pathology. Along with advances in computer image analysis, this raises the possibility for computer-assisted diagnostics in pathology to improve histopathologic interpretation and clinical care. To evaluate the potential impact of digital assistance on interpretation of digitized slides, we conducted a multireader multicase study utilizing our deep learning algorithm for the detection of breast cancer metastasis in lymph nodes. Six pathologists reviewed 70 digitized slides from lymph node sections in 2 reader modes, unassisted and assisted, with a wash-out period between sessions. In the assisted mode, the deep learning algorithm was used to identify and outline regions with high likelihood of containing tumor. Algorithm-assisted pathologists demonstrated higher accuracy than either the algorithm or the pathologist alone. In particular, algorithm assistance significantly increased the sensitivity of detection for micrometastases (91% vs. 83%, P=0.02). In addition, average review time per image was significantly shorter with assistance than without assistance for both micrometastases (61 vs. 116 s, P=0.002) and negative images (111 vs. 137 s, P=0.018). Lastly, pathologists were asked to provide a numeric score regarding the difficulty of each image classification. On the basis of this score, pathologists considered the image review of micrometastases to be significantly easier when interpreted with assistance (P=0.0005). Utilizing a proof of concept assistant tool, this study demonstrates the potential of a deep learning algorithm to improve pathologist accuracy and efficiency in a digital pathology workflow.

FIGURE 1

Schematic of reader study design. Readers review the same images in the same sequence, but with different modes: algorithm assisted, or unassisted. Readers are randomized to one of the 2 assistance “orders.” Each rectangle indicates a set of images; the color of the rectangle indicates the mode (assisted or unassisted), and the number in the rectangle indicates the number of images in that set. Readers reviewed a total of 5 images for familiarization and 70 images for formal review.

FIGURE 2

Digital image assistance by presenting algorithm predictions as a direct overlay; without assistance (A) versus with assistance (B). Suspicious regions of interest are highlighted in cyan for high confidence and green for moderate confidence (based on the algorithm predictions). In this image, both the high and moderate confidence regions of interests are confirmed as metastatic tumor as indicated by cytokeratin IHC (C). Another image example is shown in Supplemental Figure 2 (Supplemental Digital Content 1, http://links.lww.com/PAS/A677).

FIGURE 3

Improved metastasis detection with algorithm assistance. A, Data represents performance across all images by image category and assistance modality. Error bars indicate SE. The performance metric corresponds to corresponds to specificity for negative cases and sensitivity for micrometastases (micromet) and macrometastases (macromet). B, Operating point of individual pathologists with and without assistance for micrometastases and negative cases, overlayed on the receiver operating characteristic curve of the algorithm. AUC indicates area under the curve.

FIGURE 4

False-negative interpretations and false-positive algorithm predictions. A, Lymph node with >200 dispersed tumor cells but classified as negative by all readers with or without assistance; selected field of view shows the most concentrated focus of tumor cells. Left: without algorithm overlay, middle: with algorithm overlay; LYNA algorithm highlights small areas within this region with moderate confidence. Right: cytokeratin IHC as reference for presence of tumor cells; the final reference standard classification as micrometastasis for this image was reached based on counting >200 tumor cells on IHC. B, Left: stain quality and “bland” morphology led to poor visual contrast between a small macrometastasis and benign lymphoid tissue. Middle: tumor focus was outlined with moderate confidence, albeit incompletely circumscribed. Right: cytokeratin IHC for reference. Notably, despite the region highlighted by the algorithm, the tumor in this section was missed by pathologists both with and without assistance. C, Representative examples from independent cases in which LYNA falsely highlights histiocytes in the sinus (left), a giant cell (middle), and fat necrosis (right).

FIGURE 5

Average review time per image decreases with assistance. A, Average review time per image across all pathologists analyzed by category. Black circles are average times with assistance, gray triangles represent average times without assistance. Error bars indicate 95% confidence interval. B, Micrometastasis time of review decreases for nearly all images with assistance. Circles represent average review time for each individual micrometastasis image, averaged across the 6 pathologists by assistance modality. The dashed lines connect the points corresponding to the same image with and without assistance. The 2 images that were not reviewed faster on average with assistance are represented with red dot-dash lines. Vertical lines of the box represent quartiles, and the diamond indicates the average of review time for micrometastases in that modality. Micromet indicates micrometastasis; macromet, macrometastasis.