Synergistic H&E and IHC image analysis by AI predicts cancer biomarkers and survival outcomes in colorectal and breast cancer

Abstract

Background: Recent advancements in immunotherapy, particularly pembrolizumab, have shown promising results in treating metastatic colorectal cancer (CRC) and triple-negative breast cancer (TNBC). Accurate detection of predictive biomarkers, such as microsatellite instability (MSI)/mismatch repair deficiency (MMRd) and programmed death-ligand 1 (PD-L1), is key to efficacy of these treatments. Traditional methods like immunohistochemistry (IHC) and next-generation sequencing are effective but are labor intensive and require subjective interpretation.

Methods: We developed a dual-modality transformer-based model for predicting MSI/MMRd and PD-L1 status using hematoxylin & eosin and IHC stained whole slide images. We evaluated the model using area under the receiver operating curve (AUROC). Time-on-treatment (TOT) and overall survival (OS) were derived from insurance claims and analyzed by Kaplan-Meier method. Hazard ratios (HR) were determined using the Cox proportional hazard model.

Results: Our AI framework achieves clinical-grade performance, with AUROC exceeding 0.97 for MSI/MMRd prediction in CRC and 0.96 for PD-L1 prediction in breast cancer. Patients with biomarker-positive model predictions demonstrated prolonged TOT and OS when treated with pembrolizumab. For breast cancer patients, the model's predictions were superior to PD-L1 IHC in stratifying patients with improved outcomes on pembrolizumab, suggesting a reevaluation of existing PD-L1 status thresholds.

Conclusions: This study promotes the integration of advanced AI tools in clinical pathology, aiming to enhance the precision and efficiency of cancer biomarker evaluation and offering a customizable framework for varied clinical scenarios. Our model enhances predictive accuracy, integrating features from both staining methods, and exhibits superior prognostic precision compared to current biomarker assessments.

Fig. 1. Overview of the data pre-processing and model architecture.

a The pre-processing pipeline initiates with the digitization of whole slide images (WSIs), followed by tissue segmentation and tessellation of the WSIs into patches for analysis. b Illustration of the model architecture showcasing the pre-trained feature extractor, CTransPath, which processes the initial input data. c The transformer-based feature aggregation module where both hematoxylin & eosin (H&E) and immunohistochemistry (IHC) features are assimilated during model training, facilitating a comprehensive learning process that leverages the intricacies of both staining methods.

Fig. 2. Assessment of biomarker prediction performance for MSI and MMRd in CRC.

a Aggregated area under the receiver operating characteristic (AUROC) scores for predictions of microsatellite instability-high (MSI-H) and mismatch repair deficiency (MMRd) across three model types: hematoxylin & eosin (H&E) only, immunohistochemistry (IHC) only, H&E/IHC duet. Each value is the mean from a five-fold cross-validation ± 95% confidence interval (95% CI). b AUROC curves for MMRd and MSI prediction performance of the H&E/IHC duet model. c Stratified analysis of MMRd and MSI predictions, differentiated by specimen site and scanner type. d Cost-benefit analysis of duet model predictions. The X-axis indicates the False Negative Percentage (False Negatives/Total Cases). The left Y-axis represents the True Negative Percentage (True Negatives/Total Cases), and the right Y-axis denotes Sensitivity. e Visualization of attention and classification scores for deficient MMR specimens. The attention heatmap illustrates the per-patch attention rollout of our trained transformer-based feature aggregation duet model, with larger values (yellow) indicating higher contribution to the model’s prediction and smaller values (purple) indicating lower contribution. The classification heatmap displays the per-patch MMRd classification scores, with deficient MMR as the positive class and proficient MMR as the negative class. The attention x classification heatmap highlighted tiles that provide final weighted classification score.

Fig. 3. Evaluating biomarker prediction for PD-L1 status in BRCA.

a Area under the receiver operating characteristic (AUROC) curves for the prediction of PD-L1 status (where CPS ≥ 10 denotes PD-L1 positivity), comparing the performance of the hematoxylin & eosin (H&E) only, immunohistochemistry (IHC) only, and H&E/IHC duet models. b Stratified analysis of PD-L1 status predictions by specimen site and scanner type, elucidating the model’s performance across different conditions. c Cost-benefit analysis of the duet model’s predictions, with the False Negative Percentage (False Negatives/Total Cases) on the X-axis, the True Negative Percentage (True Negatives/Total Cases) on the left Y-axis, and Sensitivity on the right Y-axis. d Visualization of attention and classification scores for PD-L1-positive specimens. The attention heatmap conveys the per-patch significance through our trained transformer-based feature aggregation duet model, where yellow indicates a high contribution and purple a low contribution to the model’s output. The classification heatmap portrays the per-patch PD-L1 classification scores. The attention x classification heatmap highlighted tiles that provide final weighted classification score.

Fig. 4. Comparative survival analysis: evaluating the impact of ground truth and predicted MMRd status in CRC patients.

Kaplan–Meier survival curves for patients categorized by mismatch repair deficiency (MMRd) status, determined through pathological assessment (a), prediction from hematoxylin & eosin (H&E) whole slide images (WSIs) (b), prediction from immunohistochemistry (IHC) WSI (c), and prediction from a combined H&E/IHC WSI approach (d). The hazard ratio (HR) for the MMRd group is provided, with the mismatch repair proficient (MMRp) group serving as the reference. The shaded regions indicate 95% confidence intervals (CI). The p values were derived using the log-rank test to compare each MMRd group with the respective MMRp group.

Fig. 5. Comparative survival analysis: evaluating the impact of ground truth and predicted MSI status in CRC patients.

Kaplan–Meier survival curves for patients classified by MSI status, determined through pathological assessment (a), prediction from H&E WSI (b), prediction from IHC WSI (c), and prediction from a combined H&E/IHC WSI method (d). The HR for the MSI group is provided, using the microsatellite stable (MSS) group as the reference. Shaded areas delineate 95% confidence intervals. p values were calculated using the log-rank test to contrast each MSI group with the corresponding MSS group.

Fig. 6. Comparative survival analysis: evaluating the influence of actual and predicted PD-L1 status in patients with BRCA.

Kaplan–Meier curves comparing patient groups classified by PD-L1 status, which is determined through different methods: pathological evaluation (a), prediction using hematoxylin & eosin (H&E) whole slide images (WSIs) (b), prediction using immunohistochemistry (IHC) WSI (c), and prediction using a combined H&E/IHC WSI model (d). The hazard ratio (HR) is reported for the PD-L1-positive group with the PD-L1-negative group as the baseline for comparison. Confidence intervals (CI) at 95% are shown as shaded regions around the curves. We computed p values by employing the log-rank test to compare the survival rates of each PD-L1-positive group against their PD-L1-negative counterparts.