Prediction of recurrence risk in endometrial cancer with multimodal deep learning

Abstract

Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. Here we developed HECTOR (histopathology-based endometrial cancer tailored outcome risk), a multimodal deep learning prognostic model using hematoxylin and eosin-stained, whole-slide images and tumor stage as input, on 2,072 patients from eight EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n = 353) and two external (n = 160 and n = 151) test sets of 0.789, 0.828 and 0.815, respectively, outperforming the current gold standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low-, intermediate- and high-risk groups, respectively, by Kaplan-Meier analysis). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold standard and may help delivery of personalized treatment in EC.

Fig. 1. Overview of HECTOR.

a, Tissue segmented from the H&E WSI of EC, subsequently patched at 180 μm. A multistage vision transformer was trained using self-supervised learning by randomly sampling patches from WSIs of 1,862 patients, excluding any patients of the internal and external test sets. Patch-level features are extracted from the last eight transformer blocks. b, HECTOR taking the H&E WSI and the (FIGO 2009) anatomical stage I–III category as inputs. Extracted patch-level features are spatially and semantically averaged. The patch features are passed into both an attention-based multiple instance learning model and the im4MEC DL model (with all layers frozen), which predicts the molecular class from the H&E WSI as imPOLEmut, imMMRd, imNSMP or imp53abn. Both the anatomical stage category and image-based molecular class are fed through the Embedding layers. Gating-based attention is applied on the resulting three embeddings^,, followed by a Kronecker product for fusion. The −log(likelihood loss) was used to predict the distant recurrence-free probability function over discrete time. Risk scores were defined as the integrated predicted probabilities. MLP, multilayer perceptron; FC, Fully Connected layer.

Fig. 2. Performance of HECTOR.

a, Comparison of HECTOR performance using the C-index with alternative unimodal and two-arm DL models and CPH models fitted on clinicopathological and molecular risk factors. b, Comparison of prognostic values between HECTOR and clinicopathological and molecular risk factors combined into one risk score in a multivariable analysis. Data are presented as the HRs and 95% CIs (n = 1,254 patients). c, Residual prognostic value of all established clinicopathological and molecular risk factors when using HECTOR-predicted risk scores in a multivariable analysis. Data are presented as the HRs and 95% CIs (n = 1,254 patients). d, The 10-year distant recurrence-free probability analysis using the Kaplan–Meier method by HECTOR risk groups in the internal test set and log rank test P value. e, Experiments conducted in the LUMC external test set (n = 151 patients) with the input of multiple WSIs. f, C-index of HECTOR in the LUMC external test set randomly using one to three WSIs for all patients and repeating the experiment 100×. g, The 5-year distant recurrence-free probability analysis using the Kaplan–Meier method by HECTOR risk groups when using up to three WSIs (postaggregated by median) in the LUMC external test set and log rank test P value. GR3, grade 3; EEC, endometrioid.

Fig. 3. HECTOR explainability by analysis of HECTOR risk score with prognostic factors and analysis of input contribution.

a, Heatmap of established prognostic factors for patients included in the internal test set (n = 353 patients) ordered by predicted HECTOR risk scores. Cases with multiple alterations in POLE, MMR and/or p53 are shown. Cases lacking any of these three specific molecular alterations are considered as NSMP according to the World Health Organization 2020 classification of female genital tumors. b, Association of the prognostic factors and continuous HECTOR risk scores using multiple single linear regression with the HECTOR continuous risk scores as the dependent variable. Data are presented as the coefficients of the linear regression and 95% CIs (n = 353 patients). c, Analysis of the contribution to the HECTOR risk scores of the WSI modality in the internal test set (n = 353 patients), using the IG method. The IG values of the patches were normalized and averaged by WSI. d, IG-normalized values of the WSIs stratified by histological subtypes (top) and presence of LVSI (bottom) in the internal test set (n = 353 patients). The box plots are defined by the center tick as the median value, the lower and upper parts of the box as the first (Q1) and third (Q3) quartiles, respectively, and the bounds of whiskers are (Q1 − 1.5 × IQR, Q3 + 1.5 × IQR) where IQR is the interquartile range (Q3 − Q1). Any outlier points beyond the whiskers are displayed with point marks. e, The contribution of the image-based molecular classes to the continuous HECTOR risk score in the internal test set, using the imNSMP as the reference (ref.) group. The difference in predicted risk score is computed between the risk score given by the image-based molecular class and the one produced by using imNSMP. f, The contribution of FIGO 2009 stage to the continuous HECTOR risk score in the internal test set, using FIGO 2009 stage II as the reference group. CCC, clear cell; GR1–3, grades 1–3; SEC, serous; wt, wild-type.

Fig. 4. Morphological features contributing to HECTOR risk scores.

a, The top 5% of the regions increasing and decreasing the risk score, from the IG method, extracted for qualitative review and quantitative analysis. A representative selection of four patches for each morphological subtype (each selected from a different patient) showed the increasing risk score in the HECTOR high-risk group (right). A representative selection of four patches for each morphological subtype (each selected from a different patient) showed the decreasing risk score in the HECTOR low-risk group (left). Each patch is 180 × 180 μm². b, Among the top 5% regions, decreasing and increasing the risk score, inflammatory cells, mitotic figures and the tumor nuclei area detected and computed with DL-based image analysis tools^,. The average by patient is reported in the internal test set (n = 353). The box plots are defined by the center tick as the median value, the lower and upper parts of the box Q1 and Q3 quartiles, respectively, and the bounds of whiskers are (Q1 − 1.5 × IQR, Q3 + 1.5 × IQR). Any outlier points beyond the whiskers are displayed with point marks.

Fig. 5. Genomic and transcriptomic correlations of HECTOR risk groups using TCGA-UCEC (n = 381).

a, Analysis of the mutational frequency of the top 19 genes recognized as key oncogenic alterations in EC for each HECTOR risk group. b, Association of HECTOR risk score with the immune activation gene using multiple single linear regressions (Methods). Data are presented as the coefficients of the linear regression and 95% CIs (n = 381). c, Differential gene expression of HECTOR high-risk versus HECTOR low-risk TCGA-UCEC cases. P values of the likelihood ratio test were adjusted using the Benjamini–Hochberg FDR and statistical significance accepted <0.050.

Fig. 6. Impact of the addition of adjuvant chemotherapy to external beam radiotherapy on distant recurrence in the PORTEC-3 randomized trial by HECTOR risk group.

a, The 6-year distant recurrence-free probability by Kaplan–Meier analysis and log rank test P value shown for each HECTOR risk group stratified by randomly allocated treatment. The P value of the interaction term using categorical HECTOR risk group is shown. There was also a significant interaction between the HECTOR continuous risk scores and the treatment (P_INTERACTION = 0.014). b, For comparison with HECTOR selection, distant recurrence-free probability by Kaplan–Meier analysis from the PORTEC-3 trial for different gold standard prognostic factors in EC relying on serous histology, the FIGO 2009 stage III and the p53abn molecular class is shown. The log rank test and interaction term P values are displayed. EBRT, external beam radiotherapy; CT, chemotherapy.

Extended Data Fig. 1. Overview of the data split and downstream analyses performed in this study.

One representative WSI per patient from an Formalin-Fixed Paraffin-Embedded (FFPE) block was included. 20% of cases meeting inclusion criteria were randomly held out for an internal test set (n = 353). The remaining 80% was used for five-cross validation (n = 1,408 patients). This training dataset was enriched with dropped WSIs of FIGO 2009 stage IV cases or those with missing outcome such as the TCGA-UCEC cohort for training with self-supervised learning (n = 1,862). Two cohorts were held out as external test sets, the UMCG external test set (n = 160) and the LUMC external test set (n = 151). The LUMC external test set contains up to three FFPE blocks per case. More details for training and data split are provided in Methods. Altogether, including the two training steps and all downstream analyses, this comprehensive analysis comprised data of 2,751 tumors of women. CT, chemotherapy.

Extended Data Fig. 2. Shifts of attention scores from unimodal to multimodal model.

a, Model using only H&E WSI (unimodal) and a corresponding example of the normalized attention scores shown as overlaid on the H&E WSI as a heatmap where red is high attention score and blue low attention score. b, The two-arm model with H&E WSI and image-based molecular class predicted by im4MEC, and a corresponding example of the normalized attention scores shown as overlaid on the H&E WSI. c, The multimodal three-arm HECTOR model with H&E WSI, image-based molecular class, and stage, and a corresponding example of the normalized attention scores shown as overlaid on the H&E WSI. d, Density plot of the normalized attention scores of the heatmap shown in a,b,c for each model. e, Quantitative analysis of the distribution shift between the three models in the internal test set (n = 353 patients) using the WSI-level skewness and median of the normalized attention scores.

Extended Data Fig. 3. Morphological features increasing risk score in HECTOR high versus low risk group and quantitative spatial analysis.

a, A representative selection of four patches for each morphological subtype (each selected from a different patient) increasing the risk score in the HECTOR low risk group as compared to the features increasing the risk score in the HECTOR high risk. Each patch is 180 × 180 μm. b, Spatial analysis of top 5% regions decreasing and increasing the risk score in all WSIs of the LUMC test set based on the manually annotated areas: tumor and invasive border. (left) An example showing the annotation of the tumor area and invasive border of one WSI and heatmap showing the contribution of the regions using the IG methods. (right) The relative contribution of these two annotated areas averaged by WSI shown for each HECTOR risk group. Data are presented as the mean values and standard deviation (n = 414 WSIs).

Extended Data Fig. 4. Overview of the PORTEC-3 randomized trial and analysis of treatment response prediction by HECTOR.

In PORTEC-3, 660 evaluable patients were randomized (1:1) between adjuvant external beam radiotherapy (EBRT) alone and external beam radiotherapy in combination with concurrent and adjuvant chemotherapy (CT). For 442 patients whose WSI was available, HECTOR risk scores were inferred. HECTOR risk groups cutoffs were kept the same as the training set (Methods).