Incorporating repeated measurements into prediction models in the critical care setting: a framework, systematic review and meta-analysis

Abstract

Background: The incorporation of repeated measurements into multivariable prediction research may greatly enhance predictive performance. However, the methodological possibilities vary widely and a structured overview of the possible and utilized approaches lacks. Therefore, we [1] propose a structured framework for these approaches, [2] determine what methods are currently used to incorporate repeated measurements in prediction research in the critical care setting and, where possible, [3] assess the added discriminative value of incorporating repeated measurements.

Methods: The proposed framework consists of three domains: the observation window (static or dynamic), the processing of the raw data (raw data modelling, feature extraction and reduction) and the type of modelling. A systematic review was performed to identify studies which incorporate repeated measurements to predict (e.g. mortality) in the critical care setting. The within-study difference in c-statistics between models with versus without repeated measurements were obtained and pooled in a meta-analysis.

Results: From the 2618 studies found, 29 studies incorporated multiple repeated measurements. The annual number of studies with repeated measurements increased from 2.8/year (2000-2005) to 16.0/year (2016-2018). The majority of studies that incorporated repeated measurements for prediction research used a dynamic observation window, and extracted features directly from the data. Differences in c statistics ranged from - 0.048 to 0.217 in favour of models that utilize repeated measurements.

Conclusions: Repeated measurements are increasingly common to predict events in the critical care domain, but their incorporation is lagging. A framework of possible approaches could aid researchers to optimize future prediction models.

Fig. 1

Proposed framework for the sequential steps in the incorporation of repeated measurements in multivariable prediction. This Figure shows the proposed framework in which approaches and steps to incorporate repeated measurements in prediction research are shown. The framework consists of three domains: the observation window used to make predictions (static or dynamic), the processing of the raw data (raw data modelling, user-defined or data-driven, feature extraction and feature reduction) and explicit or implicit modeling using fixed or time-varying covariates

Fig. 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for study selection [27]

Fig. 3

Annual number of studies with repeated measurements. This Figure shows the annual number of studies with reported measurements in the critical care setting. Figure a depicts annual averages of all studies and Figure b depicts annual averages of the studies per type of analysis performed, in which single-timepoint studies do not incorporate the repeated measurements, univariable studies incorporate just one repeatedly measured variable and the included studies incorporate repeated measurements of multiple variables

Fig. 4

Comparison between analyses which do and do not include repeated measurements. This Figure shows the difference in within-study c-statistics (confidence interval) of studies which reported analyses both with and without the incorporation of repeated measurements. Abbreviations: rep = repeated measurements analysis; cs = single-timepoint analysis; no. var. = number of variables in the model