Computational pathology: A survey review and the way forward

Abstract

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Keywords: Clinical pathology; Computer aided diagnosis (CAD); Deep learning; Digital pathology; Survey; Whole slide image (WSI).

Fig. 1

We divide the data science workflow for pathology into multiple stages, wherein each brings a different level of experience. For example, the annotation/ground truth labelling stage (c) is where domain expert knowledge is consulted as to augment images with associated metadata. Meanwhile, in the evaluation phase (e), we have computer vision scientists, software developers, and pathologists working in concert to extract meaningful results and implications from the representation learning.

Fig. 2

Quality assurance and control phases developed by pathologists to oversee the clinical pathology workflow into three main phases of pre-analytical, analytical, and post-analytica phases. We further show how each of these processes can be augmented under the potential CPath applications in an end-to-end pipeline.

Fig. 3

The categorization of diagnostic tasks in computational pathology along with examples A) Detection: common detection task such as differentiating positive from negative classes like malignant from benign, B) Tissue Subtype Classification: classification task for tumorous tissue, Stroma, and adipose tissue, C) Disease Diagnosis: common disease diagnosis task like cancer staging, D) Segmentation: tumor segmentation in WSIs, and E) Prognosis tasks: shows a graph comparing survival rate and months after surgery.

Fig. 4

Distribution of diagnostic tasks in CPath for different organs from Table 9.11. This distribution includes more than 400 cited works from 2018 to 2022 inclusive. The x-axis covers different organs, the y-axis displays different diagnostic tasks, and the height of the bars along the vertical axis measures the number of works that have examined the specific task and organ. Please refer to Table 9.11 in the supplementary section for more information.

Fig. 5

WSI tissue images with different types of histological stains. Each stain highlights different areas and structures of the tissue in order to aid in visualizing underlying characteristics. Amongst this diversity, there is Hematoxylin and Eosin or H&E which is mainly used in studies as most histopathological processes can be understood from this stain. All images provided are under a Creative Commons license, specifics on the license can be found in the references.

Fig. 6

(left) shows the distribution of datasets per organ as a capture of the current trend in datasets, although the number of datasets can change over time an understanding of what organs have more available data is important for developing CAD tools. Along the vertical axis, we list different organs, while the horizontal axis shows the number of datasets; wherein the darker color denotes public availability while the light color includes unavailable or by request statuses. (right) Distribution of staining types, annotation levels, and magnification details per organ color coded consistently with the bar graph. Organs have been sorted based on the abundance of datasets. For more details, please refer to Table 9.11 in the supplementary section.

Fig. 7

Details of the five different types of annotations in computational pathology. From left to right: a) Patient-level annotations: can include high level information about the patient like status of cancer, test results, etc. b) Slide-level: are annotations associated with the whole slide, like a slide of normal tissue or a diseased one c) ROI-level annotations: are more focused on diagnosis and structure details d) Patch-level: are separated into Large FOV (field of view) and small FOV, each having different computational requirements for processing, and finally e) Pixel-level: includes information about color, texture and brightness

Fig. 8

A snapshot of the distribution of different annotation levels based on the CPath task being addressed in the surveyed literature for the purposes of highlighting the trend of datasets. The x-axis displays the different annotation levels studied in the papers (from left to right): Patient, Slide, ROI, Patch, and Pixel. The y-axis shows the different tasks (top to bottom): Detection, Diagnosis, Classification, Segmentation, and Prognosis. The height of the bars along the vertical axis measures the number of works that have examined the specific task and annotation level.

Fig. 9

Tree diagram for the optimum labeling workflow, where a CPath dataset is divided into tasks and sub-tasks based on its initial characteristics.

Fig. 10

Distribution of the most common Neural Network architectures used in the surveyed literature, based on the CPath task. The x-axis displays the Neural Network architectures used in the papers (from left to right): Custom CNN, Inception, ResNet, VGG, U-Net, and AlexNet. The y-axis shows the different tasks (top to bottom): Detection, Classification, Disease Diagnosis, Segmentation, WSI Processing, Patient Prognosis, and Others. For more details, please refer to Table 9.11 in the supplementary section.

Fig. 11

Details of types of learning using varying levels of supervision. Note that the types of tasks each type of learning can address vary based on the data that is available, as noted in the Example Task portion of the figure. However, from left to right, models trained with less supervision can still learn salient representations of the data that can be used to fine-tune models for tasks requiring more supervision. In that sense, for CPath there is a spectrum of supervision from self to strongly supervised learning that aligns well with the annotation levels shown in Fig. 7.

Fig. 12

Demonstration of the cancer statistics, featuring both the 5-Year Survival Rate and Incidence of each cancer in addition to incidence percentage of each subtype. The grey inner circle shows the incidence percentage of the respective cancer. The colored circle around each cancer corresponds to the respective 5-Year Survival Rate bin, showcasing the severity of the cancer. Darker shades (lower survival rate) means fewer people will survive the cancer after 5 years period and the cancer has poor prognosis. On the other hand, lighter shades (higher survival rates) mean more people will survive after 5 years and the cancer has good prognosis.^,, , , , , , , , , , , , , , , , ,