Deep learning informed multimodal fusion of radiology and pathology to predict outcomes in HPV-associated oropharyngeal squamous cell carcinoma

Abstract

Background: We aim to predict outcomes of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC), a subtype of head and neck cancer characterized with improved clinical outcome and better response to therapy. Pathology and radiology focused AI-based prognostic models have been independently developed for OPSCC, but their integration incorporating both primary tumour (PT) and metastatic cervical lymph node (LN) remains unexamined.

Methods: We investigate the prognostic value of an AI approach termed the swintransformer-based multimodal and multi-region data fusion framework (SMuRF). SMuRF integrates features from CT corresponding to the PT and LN, as well as whole slide pathology images from the PT as a predictor of survival and tumour grade in HPV-associated OPSCC. SMuRF employs cross-modality and cross-region window based multi-head self-attention mechanisms to capture interactions between features across tumour habitats and image scales.

Findings: Developed and tested on a cohort of 277 patients with OPSCC with matched radiology and pathology images, SMuRF demonstrated strong performance (C-index = 0.81 for DFS prediction and AUC = 0.75 for tumour grade classification) and emerged as an independent prognostic biomarker for DFS (hazard ratio [HR] = 17, 95% confidence interval [CI], 4.9-58, p < 0.0001) and tumour grade (odds ratio [OR] = 3.7, 95% CI, 1.4-10.5, p = 0.01) controlling for other clinical variables (i.e., T-, N-stage, age, smoking, sex and treatment modalities). Importantly, SMuRF outperformed unimodal models derived from radiology or pathology alone.

Interpretation: Our findings underscore the potential of multimodal deep learning in accurately stratifying OPSCC risk, informing tailored treatment strategies and potentially refining existing treatment algorithms.

Funding: The National Institutes of Health, the U.S. Department of Veterans Affairs and National Institute of Biomedical Imaging and Bioengineering.

Keywords: Deep learning; Multimodal biomarker; Oropharyngeal cancer; Pathology; Radiology.

Fig. 1

Flowchart of this study: a) multimodal data curation and annotation; b) preprocessing on pathology WSI, fragment contours (green) are generated using the CLAM toolbox, tumour annotations (red) are provided by pathologists. On radiology CT, primary tumour (yellow) and metastatic cervical lymph node annotations (blue) are provided by radiologists. c) multi-region and multiscale fusion with SwinT; d) model inference: survival and grade predictions. Red regions on CT and WSI indicate the prognostic relevant regions that the model is focusing on. W-MSA: window-based multi-head self-attention; SW-MSA: shifted window multi-head self-attention; SwinT: swin-transformer; HIPT: Hierarchical Image Pyramid Transformer.

Fig. 2

The Kaplan–Meier survival analysis of SMuRF for OPSCC DFS stratification from training set (a), validation set (b) and test set (c) and for grade classification from training set (d), validation set (e) and test set (f). Model comparisons of 7 comparable models on the test set (g) accounting for input modalities and cancer habitat regions are considered. Model comparisons using 4 different fusion schemes is performed on the test set (h).

Fig. 3

Multivariable Cox regression analysis using DFS as endpoint and multivariable logistic regression using grade as endpoint on the test set (a). Beeswarm plot of SHAP variable importance in the multivariable Cox regression analysis (b) and the multivariable logistic regression analysis (d). Mean SHAP values were converted to proportion for each variable, quantifying their contributions to the DFS predictions (c) and the grade classification (e).

Fig. 4

SMuRF histogram distributions on validation and test sets (a), and four representative patient examples of clinical information (b), cropped CT scans with primary tumour (c) and metastatic cervical lymph node annotations (e). Corresponding integrated gradient (IG) attention maps (d, f) highlighted the important regions for predictions. The IG results shown that the deep learning model focused regions within the primary tumour and metastatic cervical lymph nodes.

Fig. 5

One example high-SMuRF (a) and one example low-SMuRF (b) pathology WSIs with primary tumour annotations (red boundaries) and IG overlaid to highlight prognostic relevant regions (bottom left) detected by SMuRF. For each pathology WSI, two example 2048 × 2048 regions with corresponding attention heatmaps (c, d) within the highlighted prognostically relevant regions are provided. The attention heatmaps of HIPT model at a patch size of 256 (macro-scale) expressed differently for the high- and low-SMuRF patients: there are more condensed high attention areas related to the tumour-collagen fibre interface (c) for the high-SMuRF WSI than the low-SMuRF WSI, which contains primarily the tumour cell clusters (d). The attention heatmaps at a patch size of 16 (micro-scale) highlighted individual collagen fibre (e) for high-SMuRF while emphasized mainly the tumour cells (f) for low-SMuRF WSI. Presence of individual morphologic hallmarks (i.e., tumour-collagen fibre interface, tumour cell clusters) were evaluated and confirmed by a pathologist (T.P.).