Automated multi-scale computational pathotyping (AMSCP) of inflamed synovial tissue

Abstract

Rheumatoid arthritis (RA) is a complex immune-mediated inflammatory disorder in which patients suffer from inflammatory-erosive arthritis. Recent advances on histopathology heterogeneity of RA synovial tissue revealed three distinct phenotypes based on cellular composition (pauci-immune, diffuse and lymphoid), suggesting that distinct etiologies warrant specific targeted therapy which motivates a need for cost effective phenotyping tools in preclinical and clinical settings. To this end, we developed an automated multi-scale computational pathotyping (AMSCP) pipeline for both human and mouse synovial tissue with two distinct components that can be leveraged together or independently: (1) segmentation of different tissue types to characterize tissue-level changes, and (2) cell type classification within each tissue compartment that assesses change across disease states. Here, we demonstrate the efficacy, efficiency, and robustness of the AMSCP pipeline as well as the ability to discover novel phenotypes. Taken together, we find AMSCP to be a valuable cost-effective method for both pre-clinical and clinical research.

Fig. 1. A fine tuned 10 class model can segment relevant tissue in inflammatory arthritis.

A Mean Intersection over union (mIOU) and class frequency weighted mIOU statistics from the held-out test set for the RF and DL segmentation models at 4 different tissue granularities. Box and Whisker plots are constructed by showing the Min, 25th percentile, Median, 75th percentile and Max, each dot is one slide, n = 16. B–F Representative images of H&E (B) image, with Ground Truth (GT, C), RF (D), 9 class UNET++ (E) and Fine Tuned UNET++ (F) tissue overlays from the Test set. Bi–Fi 2x magnification of whole the joint. (Bii–Eii) 10x magnification of the anterior femoral condyle depicting synovial pannus encroachment reaching the articular cartilage. Fii 10x magnification of the posterior articular cartilage and meniscus. (Biii–Eiii) 10x magnification of the trabecular bone and bone marrow proximal to the femoral growth plate depicting an area that was difficult to predict for all models (Fiii) 10x magnification of the posterior articular cartilage and meniscus with the Fine Tuned UNET + + tissue prediction overlays.

Fig. 2. The fine tuned UNET++ model measures treatment response in the TNF-Tg with Anti-TNF therapy.

A Inferred synovial area in the held-out test set compared to historical hand drawn synovial histomorphometry area (n = 9, both Tissue inferred Area and Hand Drawn Histomorphometry Area are normally distributed, Pearson’s Correlation). B Anti-TNF therapy Study design. Six-month-old WT and TNF-Tg mice were used as controls. Eight-month-old WT or TNF-Tg mice were treated with Anti-TNF therapy or Placebo control for 6 weeks (weeks post treatment, wpt). Both left and right knees were collected and 2–3 histologic levels per knee were analyzed (n = Slides / Knees). C Using the fine tuned UNETT + + model we inferred tissue area of the Synovium (Left), Trabecular Bone (Middle) and Cartilage (Right) for 6mo WT (n = 17) and TNF Controls (n = 14) as well as Placebo (Irrelevant IgG) treated WT (n = 15), Placebo treated TNF-Tg (n = 10) and Anti-TNF treated TNF-Tg (n = 8). Each dot represents one knee (average of 2-3 histologic levels), Box and Whisker plots are construct by showing the Min, 25th percentile, Median, 75th percentile and Max. Left Panel: TNF-Tg group was not normally distributed, data was log transformed and a One-Way ANOVA with Tukey’s Post-hoc Test was performed. Middle Panel: All data was normally distributed, and a One-Way ANOVA with Tukey’s Post-hoc Test was performed. Right Panel: WT (Placebo) and TNF-Tg (Placebo) groups are not normally distributed, data was log transformed and a One-Way ANOVA with Tukey’s Post-hoc Test was performed. D Representative images (2x magnification) of 6mo WT and TNF Controls as well as 9.5 mo Placebo treated WT, Placebo treated TNF-Tg and Anti-TNF treated TNF-Tg with predicted tissue overlay. Note: black arrows denote pannus invasion of the femoral articular cartilage, the red arrows denote trabecular bone loss, and * denotes reduction in synovial area.

Fig. 3. Cell type classification model successfully identifies important cell types in inflammatory arthritis.

A Uniform Manifold Approximation and Projection (UMAP) plot after principal component analysis dimensional reduction on 856 cell features of the annotated cells (colored by cell type, n = 4,712). B A gradient boosted decision tree was trained using a parameter grid search with a nested, stratified, 5-fold cross-validation training strategy. The F1 scores (M ± SD) of the five folds for each cell class are presented with the overall weighted F1 of 0.88 ± 0.03 (M ± SD). C–E On the remaining (not-annotated) cells (~300,000) on the 9 training slides, the cell class was predicted using the most performant model. Tissue class was also predicted using the Original segmentation model. C Predicted cell class is plotted as a percent of total cells within the Fat tissue (Top) and Cartilage and Meniscus tissue (Bottom). D Representative images of Synovial tissue cell class predictions within an inflamed synovium. E Cell counts from the synovial tissue on the Healthy, Mild disease and Severe disease training slides of lymphocytes (Left), Synovial Lining Cells (Middle) and Fibroblast (Right). Each dot represents one slide, M ± SD.

Fig. 4. Computational pathology modeling recapitulates the sexual dimorphism of TNF-Tg inflammatory arthritis.

An independent set of slides from the training slides were used to validate the cell type prediction model (3 months-old: WT Male n = 5, WT Female n = 4, TNF-Tg Male n = 6, TNF-Tg Female n = 5; 4 months-old: WT Male n = 6, WT Female n = 4, TNF-Tg Male n = 5, TNF-Tg Female n = 5; 5.5 months-old: WT Male n = 4, WT Female n = 5, TNF-Tg Male n = 6, TNF-Tg Female n = 6). Tissue segmentation was first performed to segment the synovium and then cell type predictions were performed within the synovium. A Lymphocytes predictions counts. Each dot is one mouse, Box and Whisker plots are construct by showing the Min, 25th percentile, Median, 75th percentile and Max. Please note the log scale. Lymphocyte counts were found to be lognormal, a log transformation was performed on the data and a Two-Way ANOVA with Tukey’s Post-hoc test was conducted. B Synovial Lining Cell prediction counts. Each dot is one mouse, Box and Whisker plots are construct by showing the Min, 25th percentile, Median, 75th percentile and Max. Please note the log scale. A Two-Way ANOVA with Tukey’s Post-hoc test was conducted. Differences only shown between the female and male TNF-Tg mice. C Fibroblast predictions counts. Each dot is one mouse, Box and Whisker plots are construct by showing the Min, 25th percentile, Median, 75th percentile and Max. A Two-Way ANOVA with Tukey’s Post-hoc test was conducted. Differences only shown between the female and male TNF-Tg mice. D Lymphocyte predictions compare to the synovial inflammatory score as previously quantified in Figure 2F of Bell et al.. Lymphocyte counts were not normally distributed. Spearman’s correlation, TNF-Tg mice only (n = 28). Please note the log scale on the x-axis. E Synovial Lining cell counts compared to the pannus invasion score as previously quantified in Supplemental Figure 3I in Bell et al.. Synovial Lining cell counts were not normally distributed. Spearman’s correlation, TNF-Tg mice only (n = 28). Please note the log scale on the x-axis. F Total cell counts in the synovium compared to the cell area as previously quantified in Figure 2D of Bell et al.. Total cell counts and cell area in the synovium were normally distributed. Pearson’s correlation, TNF-Tg mice only (n = 28).

Fig. 5. Cell type modeling correctly classifies synovial stromal and immune cells in RA synovial biopsies.

A A subset of cells from 13 RA synovial biopsies were annotated (n = 2,341) using an active learning strategy. After nuclei detection and custom feature extraction from each cell, a gradient boosted decision tree was trained using a parameter grid search with a nested, stratified, 5-fold cross-validation training strategy. The F1 scores (M ± SD) of the five folds for each cell class are presented with the overall weighted F1 of 0.85 ± 0.01 (M ± SD). B The confusion matrix from the most performant model demonstrates the typical misclassification in this dataset (data is cell counts). Stromal cells can be mistaken for other stromal cells (vascular endothelial cells, synovial lining cells, and fibroblast) and lymphocytes can be mistaken for plasma cells. F: Fibroblast, L: Lymphoid, M/H: Macrophage/Histocyte, PC: Plasma Cell, S/C: Stromal/Connective Cell, SLC: Synovial Lining Cell; VEC: Vascular Endothelial Cell. C, D The most performant model was used to predict the cell type of the remaining cells from all RA synovial biopsies (n = 60). C Adjacent sections to the H&E-stained slides were stained with either CD3 (T-Cells), CD20 (B-Cells) and CD138 (Plasma Cells) or CLIC5 (Synovial Lining), CD3 (T-Cells), CD68 (Macrophages), and CD34 (Vascular Endothelial Cells) (n = 15). Representative images of plasma cells (Cⁱ), lymphocytes (Cⁱⁱ), synovial lining cells (Cⁱⁱⁱ) and vascular endothelial cells (C^iv) with the original H&E in the left column, prediction overlays in the middle, and adjacent slide IF in the right column. Immunostains and magnification are denoted within the image. D Correlation of machine learning predictions with quantitative histomorphometry of IF+ cells from adjacent sections of n = 15 RA synovial biopsy pieces. Top: CD138+ cells vs ML Predictions of Plasms cells; Bottom: CD3+ and CD20+ cells vs ML Predictions of Lymphocytes; Spearman’s Correlations (n = 15). Please note the log scale. E Correlation of machine learning predictions of lymphocytes (as a percent of total cells) vs the Krenn Inflammation Score, Spearman’s Correlations (n = 60). Please note the log scale on the x-axis.

Fig. 6. Cell type modeling can differentiate diffuse vs lymphoid cases with plasma cell counts alone.

A Cell type predictions were made on 58 RA biopsy specimens (n = 5 Pauci-Immune, n = 27 Diffuse, n = 26 Lymphoid) and plots of the Synovial Fibroblasts, Lymphocytes, and Plasma Cells percent of total cells demonstrate the known clinical differences among these pathotypes. Each dot is one human biopsy specimen, Box and Whisker plots are construct by showing the Min, 25th percentile, Median, 75th percentile and Max. All data was not normally distributed. Kruskal–Wallis tests with Dunn’s post hoc were performed. B Using plasma cell counts alone, we can discriminate between diffuse and lymphoid cases with a ROC-AUC of 0.82 ± 0.06 (n = 53). The optimal threshold is 0.82% plasma cells of total cells. C Representative H&E and cell type prediction overlays with low magnification and high magnification images of a pauci-immune case, diffuse case and lymphoid case.