When impact trials are not feasible: alternatives to study the impact of prediction models on clinical practice

Abstract

Patients with kidney disease have an uncertain future, with prognosis varying greatly per patient. To get a better idea of what the future holds and tailor interventions to the individual patient, prediction models can be of great value. Before a prediction model can be applied in practice, its performance should be measured in target populations of interest (i.e. external validation) and whether or not it helps improve clinical practice (i.e. whether it impacts clinical practice) should be determined. The impact would ideally be determined using an impact trial, but such a trial is often not feasible, and the impact of prediction models is therefore rarely assessed. As a result, prediction models that may not be so impactful may end up in clinical practice and impactful models may not be implemented due to a lack of impact studies. Ultimately, many prediction models end up never being implemented, resulting in much research waste. To allow researchers to get an indication of a prediction model's impact on clinical practice, alternative methods to assess a prediction model's impact are important. In this paper, we discuss several alternatives, including interviews, case-based surveys, decision comparisons, outcome modelling, before-after analyses and decision curve analyses. We discuss the general idea behind these approaches, including what information can be gathered from such studies and important pitfalls. Lastly, we provide examples of the different alternatives.

Keywords: impact; impact trial; kidney disease; nephrology; prediction model.

Figure 1:

Decision curve analysis for different decision thresholds with the Grams prediction model, an eGFR <15 mL/min/1.73 m², and the default strategies of treat all and treat none. Source: adapted from Ramspek CL et al., ‘Predicting kidney failure, cardiovascular disease and death in advanced CKD patients’ Kidney Int Rep 2022 Aug 2;7(10):2230–2241 [26]. Licensed under CC-BY 4.0. Original copyright (2022) by the International Society of Nephrology. Published by Elsevier, Inc.