Deep Learning Enabled Scoring of Pancreatic Neuroendocrine Tumors Based on Cancer Infiltration Patterns

Abstract

Pancreatic neuroendocrine tumors (PanNETs) are a heterogeneous group of neoplasms that include tumors with different histomorphologic characteristics that can be correlated to sub-categories with different prognoses. In addition to the WHO grading scheme based on tumor proliferative activity, a new parameter based on the scoring of infiltration patterns at the interface of tumor and non-neoplastic parenchyma (tumor-NNP interface) has recently been proposed for PanNET categorization. Despite the known correlations, these categorizations can still be problematic due to the need for human judgment, which may involve intra- and inter-observer variability. Although there is a great need for automated systems working on quantitative metrics to reduce observer variability, there are no such systems for PanNET categorization. Addressing this gap, this study presents a computational pipeline that uses deep learning models to automatically categorize PanNETs for the first time. This pipeline proposes to quantitatively characterize PanNETs by constructing entity-graphs on the cells, and to learn the PanNET categorization using a graph neural network (GNN) trained on these graphs. Different than the previous studies, the proposed model integrates pathology domain knowledge into the GNN construction and training for the purpose of a deeper utilization of the tumor microenvironment and its architectural changes for PanNET categorization. We tested our model on 105 HE stained whole slide images of PanNET tissues. The experiments revealed that this domain knowledge integrated pipeline led to a 76.70% test set F1-score, resulting in significant improvements over its counterparts.

Keywords: Artificial intelligence; Deep learning; Graph neural networks; Infiltration patterns; Pancreatic neuroendocrine tumors.

Fig. 1

a Diagram of infiltration patterns NI (non/minimally infiltrative), MI (moderately infiltrative), and HI (highly infiltrative). b, e Macroscopic and microscopic examples of categories IPS1/NI; c, f IPS2/MI, and d, g IPS3/HI

Fig. 2

Overview of the proposed infiltration pattern scoring pipeline. After extracting image patches from the three stain-normalized WSIs of a given case, they are fed to the CNN classifier to select representative tumor-NNP interface patches from the WSIs. Then, each patch is modeled by constructing a cell-graph on its nuclei, and its infiltration pattern score was estimated by the GNN classifier. At the end, estimated scores of all selected patches from the three WSIs are voted to predict the case-level infiltration pattern score

Fig. 3

Visual results on three example WSIs. a Original WSIs with IPS1, IPS2, and IPS3 categorization, respectively. b Pathologist annotations for the largest PanNET region in each WSI. c PanNET regions predicted by the trained CNN classifier in the first step. d UMAP projection of the features used by the CNN classifier after training it to differentiate between three classes of stroma, NNPP, and PanNET. e Example patches selected from the focus of the CNN tissue classifier

Fig. 4

a IPS predictions of the GNN classifier for patches at the tumor-NNP interface. These regions are taken from the PanNET borders exhibiting different infiltration characteristics. Note that these are not the pathologist-annotated PanNET patches, but those automatically annotated with the PanNET class by the first step CNN classifier. Thus, it may contain some incorrectly selected patches. b For an example WSI, the prediction map containing IPS predictions of patches. Likewise, these patches are those automatically annotated with the PanNET class by the first step CNN classifier. c Representative PanNET patches selected closer to (hop distance d = 0) and farther away from (d = 4 and d = 6) the tumor-NNP interface and the attention heatmaps calculated on their nodes by the Graph-GradCAM [32] visualization technique. The redder a node is, the more attention it gets from the GNN classifier