Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Abstract

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

Keywords: heterogeneity; meta-analysis; prediction; regression modeling.

Figure 1

Distribution of patient characteristics in 15 studies with 11 022 traumatic brain injury patients, after single imputation of missing values [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 2

Forest plots showing estimated multivariable logistic regression coefficients and associated 95% confidence interval per study. The largest heterogeneity was noted for pupillary reactivity (τ = 0.17) and hypotension (τ = 0.27)

Figure 3

Correlation between predictions of study‐specific models in a pairwise comparison between studies: 1‐to‐1 comparisons of predictions for all patients in the individual patient data set (n = 11 022)

Figure 4

Prediction intervals for new studies assuming a fixed effects model, random intercept model, rank = 1 model, or fully stratified model

Figure 5

Calibration plots of model developed in observational studies in a leave‐one‐study‐out cross‐validation

Figure 6

Calibration plots of model developed in RCTs in a leave‐one‐study‐out cross‐validation

Figure 7

c‐statistics leave‐one‐study‐out cross‐validation

Figure 8

Schematic representation of the research questions to be answered for the development and validation of a prediction model in an individual patient data meta‐analysis [Colour figure can be viewed at wileyonlinelibrary.com]