Analysis of 3D pathology samples using weakly supervised AI

Abstract

Human tissue, which is inherently three-dimensional (3D), is traditionally examined through standard-of-care histopathology as limited two-dimensional (2D) cross-sections that can insufficiently represent the tissue due to sampling bias. To holistically characterize histomorphology, 3D imaging modalities have been developed, but clinical translation is hampered by complex manual evaluation and lack of computational platforms to distill clinical insights from large, high-resolution datasets. We present TriPath, a deep-learning platform for processing tissue volumes and efficiently predicting clinical outcomes based on 3D morphological features. Recurrence risk-stratification models were trained on prostate cancer specimens imaged with open-top light-sheet microscopy or microcomputed tomography. By comprehensively capturing 3D morphologies, 3D volume-based prognostication achieves superior performance to traditional 2D slice-based approaches, including clinical/histopathological baselines from six certified genitourinary pathologists. Incorporating greater tissue volume improves prognostic performance and mitigates risk prediction variability from sampling bias, further emphasizing the value of capturing larger extents of heterogeneous morphology.

Keywords: 3D deep learning; 3D microscopy; 3D pathology; computational pathology; deep learning; intratumoral heterogeneity; microCT; patient prognosis; slide-free microscopy.

Figure 1:. TriPath computational workflow

(A) The 3D imaging modalities can capture high-resolution volumetric images of tissue specimens. (B) TriPath accepts raw volumetric tissue images from diverse imaging modalities as inputs. TriPath first separates the volumetric image of tissue from the background. In a common version of the workflow, the segmented volume is then treated as a stack of cuboids (3D planes) and further tessellated into smaller 3D patches (instances). (C) The patches are processed with a pretrained feature encoder network of choice, e.g., 3D convolutional neural network (CNN) or 3D Vision Transformer, leveraging transfer learning to produce a set of compact and representative features. Feature encoding with 3D CNN is illustrated in the figure. The encoded features are compressed with a domain-adapted shallow, fully-connected network. Next, an aggregator module aggregates the set of features representing all instances, automatically weighting them according to their importance towards contributing to a volume-level feature to render a patient-level prediction. TriPath also provides saliency heatmaps for clinical interpretation and validation. The computational workflow of TriPath with 2D processing is identical. Further details are described in the STAR Methods. NN, generic neural network layers dependent on the feature encoder choice; Channel C, K, intermediate channels in feature encoder; Attn, attention module; Fc1, Fc2, fully-connected layers.

Figure 2:. TriPath analysis of open-top light-sheet microscopy (OTLS) prostate cancer cohort.

The OTLS cohort contains volumetric tissue images (1μm/voxel resolution) of simulated core needle biopsies extracted from prostatectomy specimens (from the 6 regions typically targeted by urologists when performing biopsy procedures). (A) Cohort-level AUC on the development dataset (118 cancer-containing biopsies across 50 patients) for TriPath trained and tested on 3 planes separated by 20μm with the middle plane representing the largest tissue area within the biopsy (2D planes). The 3D patches within the whole volume were processed with 2D and 3D feature encoders (whole volume 2D and 3D, respectively). A clinical baseline based on the Gleason grade diagnosis of the whole prostatectomy specimen (prostatectomy grade) is also displayed. All baselines are repeated over five different experiments. (B) AUC on the held-out test dataset (53 cancer-containing biopsies across 24 patients), evaluated with models trained on the development dataset. (C) Ablation analysis with training and testing on increasing portions from the top of each volume. (D) Principal component feature space plot for 3D patches with high (unfavorable outcome), middle (no influence), and low (favorable outcome) 10% integrated gradient (IG) scores aggregated across the entire cohort. Representative 3D patches and 2D slices within each patch are displayed for each cluster. (E) 3D IG heatmap with representative 2D planes displaying unfavorable (red) and favorable (blue) prognostic regions. Statistical significance was assessed with the unpaired t-test with respect to the whole volume 3D performance. **P ≤ 0.01, ***P ≤ 0.001, and ****P ≤ 0.0001. Error bars indicate one standard deviation from the mean, over five different experiments. All scale bars are 100μm. See also Figure S2, S3, S4, S5.

Figure 3:. TriPath analysis of microcomputed tomography (microCT) prostate cancer cohort.

The microCT cohort contains volumetric tissue images of prostatectomy tissue from prostate cancer patients with 4μm/voxel resolution. (A) Cohort-level for TriPath trained and tested on the 3 planes separated by 20μm with the middle plane representing the largest tissue area within the biopsy (2D planes), the 3D patches within the whole volume processed with 2D and 3D feature encoders (whole volume 2D and 3D, respectively). A clinical baseline based on the Gleason grade diagnosis of the whole prostatectomy specimen (prostatectomy grade) is also displayed. All baselines are repeated over five different experiments. (B) Kaplan-Meier survival analysis with median BCR diagnosis date specified for each risk group, stratified at 50 percentile based on TriPath-predicted risk, for 2D planes and whole volume 3D approaches. The log-rank test was used. (C) Ablation analysis with training and testing on increasing portions from the top of each volume. (D) Principal component feature space plot for 3D patches with high (unfavorable outcome), middle (no influence), and low (favorable outcome) 10% integrated gradient (IG) scores aggregated across the entire cohort. Representative 3D patches and 2D slices within the cuboid are displayed for each cluster. (E, F) 3D IG heatmap with representative 2D planes displaying unfavorable (red) and favorable (blue) prognostic regions. Statistical significance was assessed with the unpaired t-test with respect to the whole volume 3D performance. **P ≤ 0.01 and ****P ≤ 0.0001. Error bars indicate one standard deviation from the mean, over five different experiments. All scale bars are 250μm. See also Figure S2, S3, S6, S7.

Figure 4:. Clinical validation of 3D pathology for OTLS and microCT cohorts.

TriPath is validated against clinical baselines. (A) For each biopsy sample in the OTLS cohort, 3 image slices (levels) taken from the center and ±20μm of the 3D OTLS dataset (1 μm/voxel resolution) were presented to a panel of 6 board-certified pathologists. Each pathologist provided a biopsy-level Gleason grade diagnosis. (B) Quadratic weighted kappa to assess agreement between pair of pathologists. Each point (black dot) represents an agreement between two pathologists. (C) Cohort-level ($n=50$) BCR status prediction AUCs are shown based on 6 pathologists’ diagnoses of 3 image slices (individual and consensus), the diagnosis from standard post-operative histopathology of the whole prostatectomy specimen, and TriPath-predicted risks (3D pathology). Each dot represents the cohort-level AUC repeated over five different random data splits. (D) For each biopsy block that was imaged with microCT, we obtained the adjacent tissue section and prepared an H&E-stained whole slide image (WSI). The resulting WSI and ROI (where the ROI matches the lateral field-of-view of the microCT scan) were used for risk prediction with 2D TriPath. (E) Cohort-level ($n=45$) BCR status prediction AUC based on the diagnosis of the whole prostatectomy specimen (original pathology report) and TriPath-predicted risks from H&E histology (WSI and ROI), and microCT datasets are shown. Statistical significance was assessed with the unpaired t-test with respect to TriPath performance. *P ≤ 0.05, **P ≤ 0.01, ***P ≤ 0.001, and ****P ≤ 0.0001. See also Figure S8.

Figure 5:. Plane variability analysis for open-top light-sheet microscopy (OTLS) dataset

TriPath with 2D feature encoder is trained on 2D patches from all planes of whole volume and predicts risk (the probability for the high-risk group) of individual planes of the test sample. (A) Given the plane-level predicted risks for each sample, the difference between the lower 5% and upper 95% value is computed (risk difference). (B) An arbitrary risk decision threshold (e.g., 0.5) falls within the 90% risk interval for several patients, for whom the risk group can change depending on the plane chosen for prognosis. (C) Plane-level predicted risk, which fluctuates from low-risk to high-risk, as a function of depth within the volume for a patient. (D) Principal component feature space for attention-aggregated plane-level features for the sample. The separation into two clusters along the risk group reflects the risk variation observed in (C). (E) Morphological analysis of the low-risk (depth 10) and high-risk plane (depth 275). The higher-risk plane contains a larger proliferation of tumors resembling Gleason pattern 4 than the lower-risk plane.

Figure 6:. Comparison between whole-volume and partial-volume analysis

Given the model trained on whole volume 3D, the cohort-level AUC is computed with 5-fold cross-validation for the whole volume (whole volume) or for 15% of the tissue volume randomly sampled (partial volume). For partial volume, we repeat the experiment 50 times, randomly sampling different portion of tissue volumes each time while keeping the data split the same. (A) OTLS cohort AUC spread for the partial volume analysis (teal, each dot representing an experiment) and AUC for the whole volume analysis (red). (B) IG score ranking for 3D patches when tested on the whole volume and partial volume of a given OTLS sample, where a higher ranking corresponds to a larger integrated gradient (IG) score. (C-D) The same analyses for the microCT cohort. All scale bars are 100μm.

Figure 7:. Cross-modal evaluation between OTLS and microCT cohorts

A model was trained with the whole volume 3D on one cohort and tested on the other to assess whether the model learns generalizable prostate cancer prognostic morphologies. To match the 4μm/voxel resolution and single-channel characteristics of the microCT dataset, the OTLS dataset is downsampled by a factor of 4 and only the nuclear channel is retained, resulting in a converted OTLS dataset. (A) Test AUC for the microCT cohort with TriPath trained on converted OTLS or microCT cohorts, and the cross-modal Kaplan-Meier curve for cohort stratification of high and low-risk groups. (B) Identical analyses to (A), but tested on converted OTLS with TriPath trained on microCT or converted OTLS cohorts. (C-D) Integrated gradient (IG) heatmaps for cross-modal experiments. Despite the difference in train and test modalities, TriPath identifies poorly-differentiated glands (C) and infiltrative carcinoma (D) as unfavorable prognostic morphologies, concurring with IG heatmaps from the same-modality setting. Error bars indicate one standard deviation from the mean, over five different experiments. All scale bars are 250μm.