Multimodal histopathologic models stratify hormone receptor-positive early breast cancer

Abstract

The Oncotype DX® Recurrence Score (RS) is an assay for hormone receptor-positive early breast cancer with extensively validated predictive and prognostic value. However, its cost and lag time have limited global adoption, and previous attempts to estimate it using clinicopathologic variables have had limited success. To address this, we assembled 6172 cases across three institutions and developed Orpheus, a multimodal deep learning tool to infer the RS from H&E whole-slide images. Our model identifies TAILORx high-risk cases (RS > 25) with an area under the curve (AUC) of 0.89, compared to a leading clinicopathologic nomogram with 0.73. Furthermore, in patients with RS ≤ 25, Orpheus ascertains risk of metastatic recurrence more accurately than the RS itself (0.75 vs 0.49 mean time-dependent AUC). These findings have the potential to guide adjuvant therapy for high-risk cases and tailor surveillance for patients at elevated metastatic recurrence risk.

Fig. 1. Developing a multimodal transformer model for breast cancer risk.

Early-stage breast tumors are a resected, b profiled histologically, c digitized, and d used for downstream modeling of recurrence risk. e Number of pathologic slides, pathology reports, and patients included in each split. f Histogram depicting number of slides with a given number of tiles. g Tissue detection, tessellation, transformer-based modeling of CTransPath-derived tile embeddings, pathology report scraping, tokenization and transformer-based modeling, nuclear segmentation for interpretation, tensor fusion for multimodal integration. Created in BioRender. Boehm, K. (2025) https://biorender.com/s94u717.

Fig. 2. Orpheus performance for TAILORx risk stratification.

a–c The whole-slide image (WSI)-based model reliably identifies high-risk disease (RS > 25) as defined by TAILORx across the MSK-BRCA (n = 1029), IEO-BRCA (n = 452), and MDX-BRCA (n = 572) test cohorts. d The multimodal model outperforms the WSI- and text-based unimodal models. Error bars by 1000-fold bootstrapping. e, f The multimodal model outperforms a clinicogenomic nomogram in identifying high-risk diseases. g Calibration plot and h predicted score frequencies. All results are shown for test sets.

Fig. 3. Orpheus+ performance for identifying distant recurrence.

a, b Orpheus+ risk scores versus Oncotype DX ® (ODX, N = 1464) and Multiplex DX ® (MDX, N = 575) recurrence score (RS) values for cases with or without distant recurrence. p-values by Mann–Whitney-Wilcoxon test, two-sided. c Time-dependent areas under the receiver operating characteristic curve for Orpheus+ and ODX RS scores against recurrence in the MSK-BRCA test set for cases with ODX RS ≤ 25, with associated calibration plots. The mean value represents AUC over 2–8 years. d Receiver operating characteristic curves for Orpheus+ and MDX RS scores against recurrence in the MDX-BRCA test/validation set, with associated calibration plots. **** denotes p ≤ 1e−4, *** denotes p ≤ 1e-3, ** denotes p ≤ 1e-2, and * denotes – ≤ 5e-2. In box plots, boxes denote 25th–75th percentiles with lines at the median, whiskers denote the range without outliers, and individual points denote outliers. Exact p-values, top to bottom, for a are 5e-7 and 0.80, and for b, are 0.0013 and 0.0011.

Fig. 4. Cellular and transcriptomic correlates of risk.

a Saliency map of contributory foreground tiles, with one tile-attention value pair denoted by the arrows and predicted score. b Histogram of tile-attention values. c The five highest- and lowest-attention tiles at greater magnification. d Association of cellular features with high- and low-risk tissue. Hypothesis testing was performed with the two-sided Mann–Whitney U-test with corrections for multiple testing. e High and f low relative abundance of inflammatory cells. g High and h low standard deviation of neoplastic nuclear area. i–l Quantification of stromal fraction (SF), tumor cell proliferation (Prolif), lymphocyte infiltrating signature score (LISS), and lymphocyte fraction (LF) for predicted low- and high-risk patients (50 each) depicted in blue and orange, respectively, in the MSK-BRCA cohort. p-values are generated using an independent two-sided t-test. In box plots, boxes denote 25th–75th percentiles, whiskers denote the range without outliers, and individual points denote outliers. Scale bars denote 64 µm.

Fig. 5. Potential clinical use case of the Orpheus recurrence risk prediction model.

The Orpheus multimodal prediction model for recurrence risk prediction is potentially capable of guiding decision-making for adjuvant cytotoxic chemotherapy alongside adjuvant endocrine therapy for predicted low- and high-risk patients. The model is within scope for early-stage hormone receptor-positive (HR+) and HER2- breast cancer patients. Created in BioRender. Marra, A. (2025) https://BioRender.com/l96q019.