User engagement in clinical trials of digital mental health interventions: a systematic review

Abstract

Introduction: Digital mental health interventions (DMHIs) overcome traditional barriers enabling wider access to mental health support and allowing individuals to manage their treatment. How individuals engage with DMHIs impacts the intervention effect. This review determined whether the impact of user engagement was assessed in the intervention effect in Randomised Controlled Trials (RCTs) evaluating DMHIs targeting common mental disorders (CMDs).

Methods: This systematic review was registered on Prospero (CRD42021249503). RCTs published between 01/01/2016 and 17/09/2021 were included if evaluated DMHIs were delivered by app or website; targeted patients with a CMD without non-CMD comorbidities (e.g., diabetes); and were self-guided. Databases searched: Medline; PsycInfo; Embase; and CENTRAL. All data was double extracted. A meta-analysis compared intervention effect estimates when accounting for engagement and when engagement was ignored.

Results: We identified 184 articles randomising 43,529 participants. Interventions were delivered predominantly via websites (145, 78.8%) and 140 (76.1%) articles reported engagement data. All primary analyses adopted treatment policy strategies, ignoring engagement levels. Only 19 (10.3%) articles provided additional intervention effect estimates accounting for user engagement: 2 (10.5%) conducted a complier-average-causal effect (CACE) analysis (principal stratum strategy) and 17 (89.5%) used a less-preferred per-protocol (PP) population excluding individuals failing to meet engagement criteria (estimand strategies unclear). Meta-analysis for PP estimates, when accounting for user engagement, changed the standardised effect to -0.18 95% CI (-0.32, -0.04) from - 0.14 95% CI (-0.24, -0.03) and sample sizes reduced by 33% decreasing precision, whereas meta-analysis for CACE estimates were - 0.19 95% CI (-0.42, 0.03) from - 0.16 95% CI (-0.38, 0.06) with no sample size decrease and less impact on precision. DISCUSSION: Many articles report user engagement metrics but few assessed the impact on the intervention effect missing opportunities to answer important patient centred questions for how well DMHIs work for engaged users. Defining engagement in this area is complex, more research is needed to obtain ways to categorise this into groups. However, the majority that considered engagement in analysis used approaches most likely to induce bias.

Keywords: Digital health; Digital mental health interventions; Mental health; Meta-analysis; Randomised controlled trials; Systematic review; User engagement.

Fig. 1

PRISMA flowchart for studies included in the systematic review

Fig. 2

Proportion of trials describing user engagement in methods section (A) or in results section (B) A) – How user engagement was reported in the methods section Recommended – the participant was told how to use the intervention by the study team Encouraged – reminders (e.g., notifications or emails) were sent to the participant Active User – participants meeting a pre-specified engagement level set by the study team B) – How user engagement data was reported in the results section Reported – results describe activity for at least one engagement indicator Analysis – results report an intervention effect where user engagement has been considered