A scoping review of outcome selection and accuracy of conclusions in complex digital health interventions for young people (2017-2023): methodological proposals for population health intervention research

Abstract

Background: Determining the success of population health interventions often involves assessing multiple, multidimensional outcomes rather than a single one, which presents significant methodological challenges under the evidence-based medicine paradigm. This scoping review examines outcome selection, analysis, and interpretation, and the accuracy of conclusions in complex digital health interventions promoting health among adolescents and young adults (DHI-AYA).

Methods: A comprehensive search of PubMed, EMBASE, ClinicalTrials.gov, PsycINFO, and CINAHL identified DHI-AYA implemented between 2017 and 2023. Studies were categorised by methodological choice regarding outcome hierarchical position: unique primary, multiple primary, or non-hierarchised outcomes. Outcomes were further classified into effectiveness, process, or economic categories. The authors' conclusions on intervention success were compared with conclusions drawn by the research team based on the reported outcome analysis strategy. Secondly, four analytical strategies were applied to a subset of selected interventions to illustrate the impact of outcome hierarchical position and number on conclusions about intervention success.

Results: Analysis of 100 studies linked to 26 DHI-AYA identified 251 distinct outcomes: 164 effectiveness, 78 process, and 9 economic outcomes. Seven interventions were evaluated using a unique primary outcome, 10 using multiple primary outcomes, and 9 using multiple non-hierarchised outcomes. Primary and secondary outcomes were predominantly effectiveness endpoints. The research team reclassified nine interventions (35%) deemed successful by authors as non-conclusive due to statistically conflicting results across outcomes. Most interventions deemed non-conclusive by the research team were evaluated using non-hierarchised outcomes (7/10, 70%). The choice of outcome analysis strategy substantially affected conclusions on intervention success.

Conclusions: Discrepancies in intervention success assessments highlight the need for enhanced transparency, robustness, and trustworthiness in conclusion-drawing processes. In response, five methodological proposals are formulated: (1) developing core outcome sets specific to population health intervention research (PHIR), (2) collaboratively selecting multidimensional outcomes through a steering committee that accounts for stakeholder preferences and existing theoretical models, (3) exploring multi-criteria decision analysis and consensus-driven methods to transparently combine outcomes, (4) enhancing methodological reporting through intervention development and evaluation to improve scientific integrity and reproducibility, and (5) increasing PHIR expert involvement in ethics, funding, and evaluation committees to improve recognition of evidence produced in this field.

Prospero registration number: CRD42023401979.

Keywords: Interventions; Methodological research; Outcomes; Population health intervention research; Scoping review.

Fig. 1

PRISMA flow diagram of literature search *Exclusion reasons are similar to those for full-text assessment; †These interventions were excluded because only efficacy or effectiveness evaluation protocols were available, which would have resulted in insufficient data for comprehensive analysis

Fig. 2

Description of outcome nature, hierarchical position, and statistical results by intervention category, with authors’ and researchers’ conclusions on success ^xNumber of reported outcomes excludes the 42 outcomes mentioned in protocols only. Intervention names are listed at the bottom of the figure. Intervention categorisation: A: intervention studies defining a unique primary outcome, with or without additional outcomes. B: intervention studies defining multiple primary outcomes, with or without additional outcomes. C: intervention studies defining multiple non-hierarchised outcomes. Interventions in categories A and B are displayed across the three outcome hierarchical positions—primary, secondary, and non-hierarchised—and therefore appear three times in the figure. Outcome hierarchical position: Red: primary outcomes. Yellow: secondary outcomes. Green: non-hierarchised outcomes. Grey: outcomes in protocols only (n = 42), not included in the total outcome count. Outcome statistical result: Dark shade with an asterisk (*): statistically significant outcomes (p < 0.05). Dark shade without an asterisk: not statistically significant outcomes (p ≥ 0.05). Light shade: outcomes not subjected to statistical testing. Conclusions on intervention success: S, success; F, failure; NC, non-conclusive; F†, spin (when authors reported success despite all primary outcomes being not statistically significant). Abbreviations: ICER, incremental cost-effectiveness ratio; ICUR, incremental cost-utility ratio; DALY, disability-adjusted life years; SRH, sexual and reproductive health. How to read the figure: for example, in the ARMADILLO study (category A intervention), the unique effectiveness primary outcome (attitudes) was statistically significant, while the effectiveness secondary outcome (knowledge) was not. Three exploratory (non-hierarchised) process outcomes were measured: intervention dose (statistically significant), usability and appropriateness (both not statistically tested). Both the authors and the researchers concluded that the intervention results were non-conclusive (NC)