Analysis on the Characterization of Multiphoton Microscopy Images for Malignant Neoplastic Colon Lesion Detection under Deep Learning Methods

Abstract

Background: Colorectal cancer has a high incidence rate worldwide, with over 1.8 million new cases and 880,792 deaths in 2018. Fortunately, its early detection significantly increases the survival rate, reaching a cure rate of 90% when diagnosed at a localized stage. Colonoscopy is the gold standard technique for detection and removal of colorectal lesions with potential to evolve into cancer. When polyps are found in a patient, the current procedure is their complete removal. However, in this process, gastroenterologists cannot assure complete resection and clean margins which are given by the histopathology analysis of the removed tissue, which is performed at laboratory.

Aims: In this paper, we demonstrate the capabilities of multiphoton microscopy (MPM) technology to provide imaging biomarkers that can be extracted by deep learning techniques to identify malignant neoplastic colon lesions and distinguish them from healthy, hyperplastic, or benign neoplastic tissue, without the need for histopathological staining.

Materials and methods: To this end, we present a novel MPM public dataset containing 14,712 images obtained from 42 patients and grouped into 2 classes. A convolutional neural network is trained on this dataset and a spatially coherent predictions scheme is applied for performance improvement.

Results: We obtained a sensitivity of 0.8228 ± 0.1575 and a specificity of 0.9114 ± 0.0814 on detecting malignant neoplastic lesions. We also validated this approach to estimate the self-confidence of the network on its own predictions, obtaining a mean sensitivity of 0.8697 and a mean specificity of 0.9524 with the 18.67% of the images classified as uncertain.

Conclusions: This work lays the foundations for performing in vivo optical colon biopsies by combining this novel imaging technology together with deep learning algorithms, hence avoiding unnecessary polyp resection and allowing in situ diagnosis assessment.

Keywords: Colorectal polyps; convolutional neural network; dataset; multiphoton microscopy; optical biopsy.

Figure 1

Schematic of the custom-made multimodal multiphoton microscope: tunable source; shutter (S); mirrors (M); telescope lenses (L1-L2); half wave plate; quarter wave plate; Glan-Taylor polarizer; galvanometric mirrors (GMx, GMy); scan lens (L3); tube lens (L4); objective translator; XY-translation stage (TS); dichroic mirror (D)

Figure 2

Individual image tiles acquired using two-photon fluorescence in different positions of a 30 μm thick paraffin-embedded tissue slide with sample 73 diagnosed as hyperplastic polyp. The images show cells with different shape and morphology acquired in different regions of the sample, demonstrating the capability of two-photon fluorescence for the label-free morphological assessment of tissues

Figure 3

Two-photon fluorescence image of a whole 30 μm thick paraffin-embedded tissue slide with sample 86 diagnosed as low grade adenocarcinoma. The signal originates mainly from mitochondrial NADH in the cell cytoplasm and from elastic fibers and other fluorescent molecules in the extracellular matrix. This image has been obtained by merging 37 by 29 image tiles, resulting in an overall field of view: 18.907 mm by 14.819 mm

Figure 4

Examples of tissue slides annotated by the histopathologists. Left: specimen 57 with tumoral area diagnosed as tubulovillous adenoma with high grade dysplasia; centre: specimen 73 with marked area diagnosed as hyperplastic polyp; right: specimen 86 with tumoral area diagnosed as low grade adenocarcinoma. TU stands for “tumoral”

Figure 5

Two-photon fluorescence image (bottom-left) acquired from a 30 μm thick paraffin-embedded tissue slide with sample 69-2 diagnosed as tubular adenoma with low grade dysplasia and its co-registered corresponding H&E image (top-left). Two-photon fluorescence image was obtained by concatenating 9 by 12 image tiles. The overall field of view results in 4.599 mm by 6.132 mm. Detail marked by the red box in the images on the left is represented on a magnified scale on the right. Colonic crypts and goblet cells can be identified in the H&E crop (top-right), but these features are not appreciable on the corresponding two-photon fluorescence crop (bottom-right)

Figure 6

Classification improvement with the spatially coherent predictions method versus the baseline model: correctly classified tiles are shown in green, while misclassified tiles are highlighted in red; tiles in gray are the tiles not included in the dataset. 37 tiles out of 268 were misclassified with the baseline model (left). Only 3 tiles resulted misclassified with the spatially coherent predictions method (right)

Figure 7

Effect of the confidence margin on the performance metrics: as the uncertainty margin increases (orange line) and more predictions are marked as uncertain, the performance for the rest of the predictions increases resulting in better metrics. A confidence margin of 0, 5 around the classification threshold results in a balanced accuracy of 0.9111 (0.8697 sensitivity and 0.9524 specificity) and 18.67% of the test images marked as uncertain