Clinical validation of an AI-based pathology tool for scoring of metabolic dysfunction-associated steatohepatitis

Abstract

Metabolic dysfunction-associated steatohepatitis (MASH) is a major cause of liver-related morbidity and mortality, yet treatment options are limited. Manual scoring of liver biopsies, currently the gold standard for clinical trial enrollment and endpoint assessment, suffers from high reader variability. This study represents the most comprehensive multisite analytical and clinical validation of an artificial intelligence (AI)-based pathology system, AI-based measurement of metabolic dysfunction-associated steatohepatitis (AIM-MASH), to assist pathologists in MASH trial histology scoring. AIM-MASH demonstrated high repeatability and reproducibility compared to manual scoring. AIM-MASH-assisted reads by expert MASH pathologists were superior to unassisted reads in accurately assessing inflammation, ballooning, MAS ≥ 4 with ≥1 in each score category and MASH resolution, while maintaining non-inferiority in steatosis and fibrosis assessment. These findings suggest that AIM-MASH could mitigate reader variability, providing a more reliable assessment of therapeutics in MASH clinical trials.

Fig. 1. AI-assisted workflow with representative AIM-MASH overlays and GT panel workflows.

a, In the AI-assisted workflow, the primary pathologist reviews the AIM-MASH output and does a quality control (QC) review of the Hematoxylin and Eosin (H&E) and Masson's Trichrome (MT) slides (determines whether restaining or rescanning of the slide is necessary, confirms that all trial-specific criteria are met and notes any additional findings). If the primary pathologist disagrees with any MASH component(s) by two points or more, the case goes to a review by a secondary pathologist, who independently reviews the discordant AIM-MASH score(s). If the secondary pathologist agrees with the primary pathologist’s modified score, this will be the final score; if they disagree with the primary pathologist or agree with AIM-MASH, the two pathologists will convene on a consensus call in which they agree on the final score. b, Consensus GT for each biopsy was determined by one of two panels of hepatopathologists. Each panel consisted of two main reader pathologists and an auxiliary tiebreaker pathologist. Discrepancies in scoring among the primary readers prompted the intervention of the tiebreaker pathologist, who was blind to initial assessments. When the tiebreaker’s scoring diverged from that of both primary readers, a panel discussion was convened for consensus, with the tiebreaker’s score being decisive in rare cases of continued disagreement. c, For the median GT score, when the tiebreaker’s scoring diverged from that of both primary readers, the median of the three scores was considered final. Overall, five distinct pathologists contributed to establishing the GT.

Fig. 2. Scanner repeatability and reproducibility of AIM-MASH.

a, For scanner repeatability, a subset of 150 cases from the clinical validation were scanned multiple times using the same Leica Aperio AT2 scanner at ×40 magnification on three nonconsecutive days (intrasite, interscan). b, For scanner reproducibility, the same slides were scanned once at three different laboratories by three different operators using three different Leica Aperio AT2 scanners at ×40 magnification (intersite). Bootstrap percentile P values showing statistical significance for the one-sided hypothesis that the mean agreement rate between algorithm scores for each scan is greater than 0.85 are as follows: ***P < 0.0001; **P < 0.01; *P < 0.05; not significant (NS), P ≥ 0.05. Whiskers show the 95% CIs for mean agreement rate estimated using 2,000 bootstraps. Dashed lines indicate 85% agreement.

Fig. 3. Accuracy concordance comparison of MASH histologic components and comparisons for MASH aggregate component scores (F2 and F3 versus other and NAS ≥ 4 with ≥1 in each score category versus other) and MASH resolution.

a,b, Accuracy comparison, based on linearly WK, between AIM-MASH (without pathology review) versus GT and IMR versus GT in a and between AI assisted (AIM-MASH with pathology review) versus GT and IMR versus GT in b for MASH components. c, Accuracy comparison, based on kappa, between AI assisted versus GT and IMR versus GT for aggregate components relevant to clinical trial enrollment and endpoint criteria, including the score-based enrollment requirement, MAS ≥ 4 with a score of at least one for each component, fibrosis score of 2 or 3, and the NASH resolution endpoint, defined as a ballooning score of 0, a lobular inflammation score of 0 or 1 and any score for steatosis. Point estimates are shown on top of each bar, with whiskers representing the 95% CIs estimated from 2,000 bootstrap samples. Non-inferiority (NI) was assessed using bootstrap percentile P values for testing the one-sided hypothesis that the LB of the 95% CIs of the difference in AIM-NASH versus GT or AI assisted versus GT and IMR versus GT is not smaller than −0.1. S (superiority) was assessed by testing the one-sided hypothesis that the LB of the difference is greater than 0. ***P < 0.0001; **P < 0.01; *P < 0.05; NS, P ≥ 0.05. ‘+’ in c indicates aggregate components where the LB of the 95% CIs for AI assisted versus GT kappa is greater than the upper bound of the IMR versus GT kappa.

Fig. 4. WK analysis for MASH components AI assisted and median panel comparisons.

The same cohort of 1,481 cases used in analytical and clinical validation was used to determine the accuracy of AI-assisted reads against two panels of readers. Median GT (panel 1, using median scores, described in Fig. 1c), instead of panel calls for consensus and median IMR (panel 2), derived from a minimum of three IMRs, was determined. AI-assisted scores for each component met the non-inferiority performance criteria described in Statistical analysis (Methods). Superiority was not observed for any of the components. Whiskers represent 95% CIs estimated using 2,000 bootstrap samples. ***P < 0.0001; **P < 0.01; *P < 0.05; NS, P ≥ 0.05.