Predictive Modeling of Heterogeneous Treatment Effects in RCTs: A Scoping Review

Abstract

Importance: The Predictive Approaches to Treatment Effect Heterogeneity (PATH) Statement of 2020 proposed predictive modeling for identifying heterogeneity in treatment effects (HTE) in randomized clinical trials (RCTs). It described 2 approaches: risk modeling, which develops a multivariable model predicting individual baseline risk of study outcomes and then examines treatment effects across strata of predicted risk, and effect modeling, which develops a model that directly predicts individual treatment effects using a variety of regression and machine learning methods.

Objective: To identify, describe, and evaluate findings from reports that cited the PATH Statement and presented predictive modeling of HTE in RCTs.

Evidence review: Reports were identified using PubMed, Google Scholar, Web of Science, and SCOPUS through July 5, 2024. Using double review with adjudication, reports were assessed for consistency with PATH Statement recommendations, credibility of HTE findings (applying criteria adapted from the Instrument to Assess Credibility of Effect Modification Analyses), and clinical importance of credible findings.

Findings: A total of 65 reports (presenting 31 risk models and 41 effect models) analyzing 162 RCTs were identified, with credible, clinically important HTE in 24 reports (37%). Contrary to PATH Statement recommendations, only 25 of 48 studies with positive overall findings included a risk model. Most effect models were exploratory, including multiple predictors with little prior evidence for HTE. Claims of HTE were noted in 23 risk modeling and 31 effect modeling reports but were more likely to meet credibility criteria with risk modeling (20 of 23 reports [87%]) than effect modeling (10 of 31 reports [32%]). For effect modeling, validation of HTE findings in external datasets was critical in establishing credibility. Credible HTE from either approach was usually judged clinically important (24 of 30 reports [80%]). In the 19 reports from RCTs suggesting overall treatment benefits, modeling identified subgroups of 5% to 67% of patients predicted to experience no benefit or net treatment harm. In the 5 reports that found no overall benefit, subgroups of 25% to 60% of patients were nevertheless predicted to benefit.

Conclusions and relevance: This scoping review of 65 reports of multivariable predictive modeling of HTE in RCTs identified credible, clinically important HTE in 37%. Risk modeling was more likely than effect modeling to find credible HTE, but external validation of HTE findings served to increase the credibility of findings from exploratory effect models.

Figure 1.. Flowchart for Identification and Screening of Reports

All reports considered cited the Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement. Reports were excluded if they did not present a predictive model of individual treatment effects from randomized clinical trial (RCT) data. IPDMA indicates independent patient data meta-analysis. ^aDetails of these 16 studies are presented in eTable 2 in Supplement 1.

Figure 2.. Adjudicated Results of Review of Eligible Reports

Type of predictive modeling (risk or effect), claims by authors of heterogeneity of treatment effects (HTE), assessed credibility of HTE (using adapted Instrument to assess Credibility of Effect Modification Analyses criteria), and assessed clinical importance of HTE (using Predictive Approaches to Treatment effect Heterogeneity Statement definition of clinical importance). EM indicates effect modification; RM, risk modification.