Readdressing the Ongoing Challenge of Missing Data in Youth Ecological Momentary Assessment Studies: Meta-Analysis Update

Abstract

Background: Ecological momentary assessment (EMA) is pivotal in longitudinal health research in youth, but potential bias associated with nonparticipation, omitted reports, or dropout threatens its clinical validity. Previous meta-analytic evidence is inconsistent regarding specific determinants of missing data.

Objective: This meta-analysis aimed to update and expand upon previous research by examining key participation metrics-acceptance, compliance, and retention-in youth EMA studies. In addition, it sought to identify potential moderators among sample and design characteristics, with the goal of better understanding and mitigating the impact of missing data.

Methods: We used a bibliographic database search to identify EMA studies involving children and adolescents published from 2001 to November 2023. Eligible studies used mobile-delivered EMA protocols in samples with an average age up to 18 years. We conducted separate meta-analyses for acceptance, compliance, and retention rates, and performed meta-regressions to address sample and design characteristics. Furthermore, we extracted and pooled sample-level effect sizes related to correlates of response compliance. Risk of publication bias was assessed using funnel plots, regression tests, and sensitivity analyses targeting inflated compliance rates.

Results: We identified 285 samples, including 17,441 participants aged 5 to 17.96 years (mean age 14.22, SD 2.24 years; mean percentage of female participants 55.7%). Pooled estimates were 67.27% (k=88, 95% CI 62.39-71.96) for acceptance, 71.97% (k=216, 95% CI 69.83-74.11) for compliance, and 96.57% (k=169, 95% CI 95.42-97.56) for retention. Despite overall poor moderation of participation metrics, acceptance rates decreased as the number of EMA items increased (log-transformed b=-0.115, SE 0.036; 95% CI -0.185 to -0.045; P=.001; R²=19.98), compliance rates declined by 0.8% per year of publication (SE 0.25, 95% CI -1.3 to -0.3; P=.002; R²=4.17), and retention rates dropped with increasing study duration (log-transformed b=-0.061, SE 0.015; 95% CI -0.091 to 0.032; P<.001; R²=10.06). The benefits of monetary incentives on response compliance diminished as the proportion of female participants increased (b=-0.002, SE 0.001; 95% CI -0.003 to -0.001; P=.003; R²=9.47). Within-sample analyses showed a small but significant effect indicating higher compliance in girls compared to boys (k=25; g=0.18; 95% CI 0.06-0.31; P=.003), but no significant age-related effects were found (k=14; z score=0.05; 95% CI -0.01 to 0.16).

Conclusions: Despite a 5-fold increase in included effect sizes compared to the initial review, the variability in rates of missing data that one can expect based on specific sample and design characteristics remains substantial. The inconsistency in identifying robust moderators highlights the need for greater attention to missing data and its impact on study results. To eradicate any health-related bias in EMA studies, researchers should collectively increase transparent reporting practices, intensify primary methodological research, and involve participants' perspectives on missing data.

Trial registration: PROSPERO CRD42022376948; https://www.crd.york.ac.uk/PROSPERO/view/CRD42022376948.

Keywords: adolescents; ambulatory assessment; children; dropout; ecological momentary assessment; experience sampling methodology; meta-analysis; missing data; mobile devices; youth.

Figure 1

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 flow diagram. The literature search combined studies selected by the previous edition of the meta-analysis via identification of new studies via databases and registers and identification of new studies via targeted backward chasing of systematic reviews and meta-analyses. Unique studies identified via databases and those identified via other methods do not strictly sum up due to multiple publications on same samples. We adapted the PRISMA 2020 flow diagram [48] by adding further detail on reasons for exclusion at the level of title-abstract screening.

Figure 2

Meta-regression coefficients of sample and design characteristics for participation metrics. Gray and black error bars correspond to the SE and 95% CI, respectively. The dashed line represents the null effect of regression coefficients. Acceptance and retention rates were arcsine transformed before plotting on the natural scale. In addition, moderator variables with skew >3.0 were log-transformed. For binary and categorical predictors, the category expressing absence of the respective sample or design characteristics was used as the reference category. Undepicted estimates were not calculated due to low cell frequencies or overall missingness. ACT: active forms of participant care; CG: clinical groups; EMA: ecological momentary assessment; HC: healthy controls; IN: inpatient; MIN: minimal forms of participant care; NNR: none or not reported; OUT: outpatient; PA: parents assisting their children’s participation; PE: parallel parent ecological momentary assessment; PPP: pre, peri, or posttreatment; PR: sporadic parent reports; PSY: psychiatric disorders; SOM: somatic diseases; VG: visual enhancement or gamification.

Figure 3

Forest plot of gender differences in response compliance across studies. The size of the square corresponds to the relative weight of the analysis in the meta-analytic random-effects model. Error bars show the 95% CI for each sample estimate. Point estimates below 0 provide support for boys being more compliant than girls, whereas point estimates above 0 provide support for girls being more compliant than boys. The diamond below the sample-specific part of the forest plot depicts the pooled estimate and its width marks the corresponding CI. A pooled effect size is considered significant when the diamond does not cross the dashed line at 0 [79,84,89,91,92,109,127,131,140,151,182,195,201,214,227,239,240, 242,246,261,265,283,289]. ANX: anxiety disorders; DYN: dynamic incentivization scheme; FLAT: flat incentivization scheme; HC: healthy controls.

Figure 4

Forest plot of age-compliance correlation coefficients across studies. The size of the square corresponds to the relative weight of the analysis in the meta-analytic random-effects model. Error bars show the 95% CI for each sample estimate. Diverse types of correlation coefficients were harmonized as Fisher’s z scores to facilitate quantitative synthesis. The diamond below the sample-specific part of the forest plot depicts the pooled estimate and its width marks the corresponding 95% CI. A pooled effect size is considered significant when the diamond does not cross the dashed line at 0 [84,92,140,189,195,201,214,227,240,246,265,283,297,302].