Mining the interpretable prognostic features from pathological image of intrahepatic cholangiocarcinoma using multi-modal deep learning

Abstract

Background: The advances in deep learning-based pathological image analysis have invoked tremendous insights into cancer prognostication. Still, lack of interpretability remains a significant barrier to clinical application.

Methods: We established an integrative prognostic neural network for intrahepatic cholangiocarcinoma (iCCA), towards a comprehensive evaluation of both architectural and fine-grained information from whole-slide images. Then, leveraging on multi-modal data, we conducted extensive interrogative approaches to the models, to extract and visualize the morphological features that most correlated with clinical outcome and underlying molecular alterations.

Results: The models were developed and optimized on 373 iCCA patients from our center and demonstrated consistent accuracy and robustness on both internal (n = 213) and external (n = 168) cohorts. The occlusion sensitivity map revealed that the distribution of tertiary lymphoid structures, the geometric traits of the invasive margin, the relative composition of tumor parenchyma and stroma, the extent of necrosis, the presence of the disseminated foci, and the tumor-adjacent micro-vessels were the determining architectural features that impacted on prognosis. Quantifiable morphological vector extracted by CellProfiler demonstrated that tumor nuclei from high-risk patients exhibited significant larger size, more distorted shape, with less prominent nuclear envelope and textural contrast. The multi-omics data (n = 187) further revealed key molecular alterations left morphological imprints that could be attended by the network, including glycolysis, hypoxia, apical junction, mTORC1 signaling, and immune infiltration.

Conclusions: We proposed an interpretable deep-learning framework to gain insights into the biological behavior of iCCA. Most of the significant morphological prognosticators perceived by the network are comprehensible to human minds.

Keywords: Cholangiocarcinoma; Deep learning; Interpretable model; Prognosis; Tumor morphology.

Fig. 1

Classification networks and global segmentation map. A Representation of annotated WSIs. B Representation of labeled tiles for training the classification networks. C Normalized confusion matrices of the classification results. D Area under the curve (AUC) of each tissue category of the classification networks. E Two examples of the global segmentation map (GSM) activated by class for WSI. TT-p was masked in yellow, TT-s was masked in blue, HN was masked in brown, and TLS was highlighted in green. WSI, whole slide image; LT, peri-tumor liver tissue; TT, tumor tissue; TT-p, tumor parenchyma; TT-s, tumor stroma; TLS, tertiary lymphoid structure; HN, hemorrhage and necrosis

Fig. 2

Evaluation of the predictive performance under various conditions. A C-indices via the minimal (GSmin), maximal (GSmax), mean (GSmean), and standard deviation (GSsd) of the GSs among patients with multiple WSIs in cohorts T and V1. B C-indices via TiRS according to different sampling methods. C C-indices via TiRS according to different tile counts under different magnification scales. D C-indices via GS, TiRS, CRS, and clinical index in cohorts T, V1, and V2. E Kaplan–Meier curves of survival for high and low CRS. The bars represent the 95% confidence intervals. *P < 0.05; **P < 0.01; ***P < 0.001; GS, GSM score; WSI, whole slide image; TiRS, tile risk score; CRS, consensus risk score; GSM, global segmentation map

Fig. 3

Deconstruction and visualization of architectural features. A Visualization of high-risk (masked in red) and low-risk (masked in blue) features attended by prognostic model 1 using occlusion sensitivity map (OSM). Each WSI was demonstrated in three forms: the original global segmentation map (left), the OSM (middle), and the merged image (right). The first panel showed the risk differences according to the presence and distribution of TLS (tTLS and pTLS were indicated by green circle and black circled curve respectively). The second panel highlighted the prognostic significance of the smoothness of invasive margin (blue circled curve). The third panel revealed the opposite prognostic impacts of TT-p (yellow) and TT-s (dark blue). The fourth, fifth, and sixth panels demonstrated other significant architectural features, including the presence of necrosis (brown circled curve), disseminated foci (orange circle), and adjacent micro-vessels (red circled curve). B The prognostic impacts of predefined architectural parameters in cohorts T, V1, and V2. Red circles indicate hazard ratios greater than 1, while blue circles indicate hazard ratios less than 1. The size of the circles indicates the P value. The area ratio of pTLS to LT, the area ratios of HN and TT-s to TT, the distribution variance of HN, and the smoothness of invasive margin showed potential associations with high risk, while the area ratios of tTLS and TT-p to TT and the distribution variance of tTLS were associated with low risk. WSI, whole slide image; TLS, tertiary lymphoid structure; tTLS, intra-tumor TLS; pTLS, peri-tumor TLS; TT, tumor tissue; LT, peri-tumor liver tissue; HN, hemorrhage and necrosis region; TT-p, tumor parenchyma; TT-s, tumor stroma

Fig. 4

Quantified morphological analysis revealed fine-grained features. A Significant associations were found between TiRS and iCCA subtypes. B Poorly differentiated iCCAs had significantly higher TiRS than well/moderately differentiated tumors. The bars represent the 95% confidence intervals. C Representations of tiles from WSIs with high or low TiRS, tumor cell nuclei were automatically segmented by CellProfiler (masked in dark green and demonstrated on the right of each original tile). Tumor nuclei from high TiRS tiles exhibited larger size, more distorted shape, while their nuclear envelope and textural contrast were less prominent. D The raw profiles containing all 732 measurements of tumor cell nuclei were processed by dimensionality reduction. E The names and coefficients of the most significant measurements that correlated with TiRS after lasso regression. Red color indicated positive correlation and blue color indicated negative correlation. *P < 0.05; **P < 0.01; ***P < 0.001; TiRS, tile risk score; iCCA, intrahepatic cholangiocarcinoma; WSI, whole slide image

Fig. 5

Relevance of TiRS to molecular alterations. A Correlation of TiRS with hallmark gene sets of cancer. The upper panel was based on transcriptomics data, and the lower panel was based on proteomics data. B Relevance of TiRS to tumor immune infiltration. C Comparison of tiles with opposing gene set scores that were morphologically discernible. Infiltrating immune cells were marked in green. TiRS, tile risk score