Promising algorithms to perilous applications: a systematic review of risk stratification tools for predicting healthcare utilisation

Abstract

Objectives: Risk stratification tools that predict healthcare utilisation are extensively integrated into primary care systems worldwide, forming a key component of anticipatory care pathways, where high-risk individuals are targeted by preventative interventions. Existing work broadly focuses on comparing model performance in retrospective cohorts with little attention paid to efficacy in reducing morbidity when deployed in different global contexts. We review the evidence supporting the use of such tools in real-world settings, from retrospective dataset performance to pathway evaluation.

Methods: A systematic search was undertaken to identify studies reporting the development, validation and deployment of models that predict healthcare utilisation in unselected primary care cohorts, comparable to their current real-world application.

Results: Among 3897 articles screened, 51 studies were identified evaluating 28 risk prediction models. Half underwent external validation yet only two were validated internationally. No association between validation context and model discrimination was observed. The majority of real-world evaluation studies reported no change, or indeed significant increases, in healthcare utilisation within targeted groups, with only one-third of reports demonstrating some benefit.

Discussion: While model discrimination appears satisfactorily robust to application context there is little evidence to suggest that accurate identification of high-risk individuals can be reliably translated to improvements in service delivery or morbidity.

Conclusions: The evidence does not support further integration of care pathways with costly population-level interventions based on risk prediction in unselected primary care cohorts. There is an urgent need to independently appraise the safety, efficacy and cost-effectiveness of risk prediction systems that are already widely deployed within primary care.

Keywords: Decision Support Systems, Clinical; Electronic Health Records; Primary Health Care; Public Health; Safety Management.

Figure 1

An infographic describing an idealised process for developing and deploying a risk prediction tool within a healthcare system. In black is the deployment cycle, linking risk prediction tools and their associated population health management measures to a lifecycle of evidence generation, impact evaluation and monitoring for negative consequences that are fed back into the model and intervention.

Figure 2

A PRISMA flow diagram showing the process of study selection for our analysis. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Figure 3

An infographic summarising the validation characteristics of the identified models that underwent external validation or real-world testing. Models that underwent more extensive validation processes are represented by larger boxes. Each box contains aggregated data for all of the external validation and real-world evaluation studies for each model. Validation countries are represented by flags with the number of studies based in each country overlying. R² and C-statistics are displayed as ranges for all of the outcome measures tested for each model for illustrative purposes only. A&E, accident and emergency department; PPV, positive predictive value; RCT, randomised controlled trial; RR, risk ratio.