Identifying factors associated with user retention and outcomes of a digital intervention for substance use disorder: a retrospective analysis of real-world data

Abstract

Objectives: Successful delivery of digital health interventions is affected by multiple real-world factors. These factors may be identified in routinely collected, ecologically valid data from these interventions. We propose ideas for exploring these data, focusing on interventions targeting complex, comorbid conditions.

Materials and methods: This study retrospectively explores pre-post data collected between 2016 and 2019 from users of digital cognitive behavioral therapy (CBT)-containing psychoeducation and practical exercises-for substance use disorder (SUD) at UK addiction services. To identify factors associated with heterogenous user responses to the technology, we employed multivariable and multivariate regressions and random forest models of user-reported questionnaire data.

Results: The dataset contained information from 14 078 individuals of which 12 529 reported complete data at baseline and 2925 did so again after engagement with the CBT. Ninety-three percent screened positive for dependence on 1 of 43 substances at baseline, and 73% screened positive for anxiety or depression. Despite pre-post improvements independent of user sociodemographics, women reported more frequent and persistent symptoms of SUD, anxiety, and depression. Retention-minimum 2 use events recorded-was associated more with deployment environment than user characteristics. Prediction accuracy of post-engagement outcomes was acceptable (Area Under Curve [AUC]: 0.74-0.79), depending non-trivially on user characteristics.

Discussion: Traditionally, performance of digital health interventions is determined in controlled trials. Our analysis showcases multivariate models with which real-world data from these interventions can be explored and sources of user heterogeneity in retention and symptom reduction uncovered.

Conclusion: Real-world data from digital health interventions contain information on natural user-technology interactions which could enrich results from controlled trials.

Keywords: digital health intervention; real-world data exploration; real-world uptake; secondary use; substance use disorder.

Figure 1.

Association of participant retention with pre-engagement recovery progression and sociodemographic characteristics. Association analysis showing (A) odds ratios of participant retention against 4 clinical outcome measures; (B) odds ratios of association with sociodemographic membership. Reference groups are male gender, White ethnicity, community environment, and alcohol as a target substance. ORs are shown with their 95% confidence intervals.

Figure 2.

Associations of participant recovery progression measured on item and aggregate levels with assessment time pre- and post-BFO engagement, and gender. Odds ratios (reference groups: pre-engagement, male, and pre-engagement male) are shown with their 95% confidence intervals. The PHQ-4 and the WHOQOL-BREF are shown with affiliated items.

Figure 3.

Subgroup questionnaire response illustration. Stacked bar plots showing pre- and post-BFO engagement differences in responses between men and women on a selection of questionnaire items and in terms of exceeding the cutoff for anxiety and depression. Each color block reflects a different categorical level for the outcome displayed. An asterisk indicates statistical significance of the interaction between assessment time and gender for (A), and statistical significance of the difference between men and women for (B), suggested by regression models.

Figure 4.

Averaged receiver operating characteristic curves for binary outcomes. The curve shows the averaged prognostic performance of random forests in 100 validation datasets. AUC${}_{\text{retention}}$ = 0.59, AUC${}_{\text{anxiety}}$ = 0.78, AUC${}_{\text{depression}}$ = 0.79, AUC${}_{\text{alcohol dependence}}$ = 0.77, AUC${}_{\text{drug dependence}}$ = 0.74.

Figure 5.

Accumulated local effects of 3 features on the prediction of anxiety. The predicted probability of post-engagement anxiety increased with pre-engagement levels of (A) joylessness, (B) impact of emotions, and (C) negative self-perception.

Figure 6.

Associations of repeated misclassification post-engagement with pre-engagement participant characteristics. Reference groups are male gender, white ethnicity, community environment, alcohol as a target substance for associations with the misclassification of post-engagement (A) anxiety, (B) depression, and (C) alcohol dependence, and alcohol and drugs as target substances for associations with the misclassification of post-engagement (D) drug dependence. ORs are shown with their 95% confidence intervals.