Identification of key factors related to digital health observational study adherence and retention by data-driven approaches: an exploratory secondary analysis of two prospective longitudinal studies

Abstract

Background: Longitudinal digital health studies combine passively collected information from digital devices, such as commercial wearable devices, and actively contributed data, such as surveys, from participants. Although the use of smartphones and access to the internet supports the development of these studies, challenges exist in collecting representative data due to low adherence and retention. We aimed to identify key factors related to adherence and retention in digital health studies and develop a methodology to identify factors that are associated with and might affect study participant engagement.

Methods: In this exploratory secondary analysis, we used data from two separate prospective longitudinal digital health studies, conducted among adult participants (age ≥18 years) during the COVID-19 pandemic by the BIG IDEAs Laboratory (BIL) at Duke University (Durham, NC, USA; April 2, 2020 to May 25, 2021) and Evidation Health (San Mateo, CA, USA; April 4 to Aug 31, 2020). Prospective daily or weekly surveys were administered for up to 15 months in the BIL study and daily surveys were administered for 5 months in the Evidation Health study. We defined metrics related to adherence to assess how participants engage with longitudinal digital health studies and developed models to infer how demographic factors and the day of survey delivery might be associated with these metrics. We defined retention as the time until a participant drops out of the study. For the purpose of clustering analysis, we defined three metrics of survey adherence: (1) total number of surveys completed, (2) participation regularity (ie, frequency of filling out surveys consecutively), and (3) time of activity (ie, engagement pattern relative to enrolment time). We assessed these metrics and explored differences by age, sex, race, and day of survey delivery. We analysed the data by unsupervised clustering, survival analysis, and recurrent event analysis with multistate modelling, with analyses restricted to individuals who provided data on age, sex, and race.

Findings: In the BIL study, 5784 unique participants with the required demographic data completed 388 600 unique daily surveys (mean 67 [SD 90] surveys per participant). In the Evidation Health study, 89 479 unique participants with the required demographic data completed 2 080 992 unique daily surveys (23 [32] surveys per participant). Participants were grouped into adherence clusters based on the three metrics of adherence, and we identified statistically discernible differences in age, race, and sex between clusters. Most of the individuals aged 18-29 years were observed in the clusters with low or medium adherence, whereas the oldest age group (≥60 years) was generally more represented in clusters with high adherence than younger age groups. For retention, survival analysis indicated that 18-29 years was the age group with the highest risk of exiting the study at any given point in time (BIL study, hazard ratio [HR] for 18-29 years vs ≥60 years, 1·69 [95% CI 1·53-1·86; p<0·0001]; Evidation Health study, HR 1·50 [1·47-1·53; p<0·0001]). Sex and race were not discernible predictors of retention in the BIL study. In the Evidation Health study, male participants (vs female participants; HR 0·96 [0·94-0·98]; p<0·0001) and White participants (vs Asian participants; HR 0·96 [0·93-0·98; p=0·0004) had a lower risk of study exit, and Other race participants (vs Asian participants) had a higher risk of study exit (HR 1·10 [1·06-1·14; p<0·0001]). Recurrent event analysis confirmed age as the factor most associated with adherence; for the 18-29 years age group (vs ≥60 years group), the transition intensity from an active to inactive state per day in the BIL study was 1·661 (95% CI 1·606-1·718) and in the Evidation Health study was 1·108 (1·094-1·121). Participation patterns were variable by race and sex between the studies.

Interpretation: Our analyses revealed that age was consistently associated with adherence and retention, with younger participants having lower adherence and higher dropout rates than older participants. Unsupervised clustering and survival analyses are established methods in this field, whereas the use of recurrent event analysis, was, to our knowledge, the first instance of the application of this method to remote digital health data. These methods can help to understand participant engagement in digital health studies, supporting targeted measures to improve adherence and retention.

Funding: US National Science Foundation, US National Institutes of Health, Microsoft AI for Health, Duke Clinical and Translational Science Institute, North Carolina Biotechnology Center, Duke MEDx, Duke Bass Connections, Duke Margolis Center for Health Policy, and Duke Office of Information Technology.

Figure 1:. Proposed method to achieve target demographics from enrolment through to final data analysis

(A) Graphical representation of the demographics of participants enrolling in a conventional longitudinal digital health study and the subsequent study population generating the data for analysis. (B) By analysing participant engagement during the study, research teams can undertake proactive measures to mitigate challenges with participant adherence and retention.

Figure 2:. Adherence to digitally delivered surveys

(A) An example participant’s survey filling patterns over the course of a study. ACT represents the days surveys were completed, INACT represents the days surveys were not completed, and EXIT represents the days after the participant left the study; grey dashes represent days. The network model on the right shows possible paths between states as represented by the arrowhead direction. Once a participant exits the study, there is no path to re-enter. (B) Heatmap showing clustering of daily survey filling patterns in the BIL study. Due to the study design changing from a weekly to daily survey method, the number of days extended beyond a 12-month period. (C) Heatmap showing clustering of daily survey filling patterns in the Evidation Health study. In the heatmaps, each column represents one day and each row represents one participant. Unclustered maps are presented in appendix 1 (p 5). (D) Adherence characterisation of the different clusters for both studies (left, BIL study; right, Evidation Health study). Participants within each cluster were ordered by their study ID number. BIL=Duke University BIG IDEAs Laboratory.

Figure 3:. Study participants clustered on daily survey filling patterns and stratified by demographic factors

Plots show the percentages of each survey filling pattern cluster for participants in each study separated by race, age, and sex. BIL=Duke University BIG IDEAs Laboratory.

Figure 4:. Kaplan–Meier curves by age, race, and sex

Kaplan–Meier curves showing the changing retention probability over time in demographic groups in each study, with the event defined as participants who had exited the study (EXIT state). Numbers at risk, numbers censored, and log-rank p values are provided in appendix 1 (pp 13-15). BIL=Duke University BIG IDEAs Laboratory.

Figure 5:. Multistate model transition intensities by race, age, sex, and day of survey delivery

Transition intensities ordered by INACT to ACT (INACT–ACT) and ACT to INACT (ACT–INACT) for different covariates in the multistate model for each study, where ACT represents an active state and INACT represents an inactive state. The BIL model also included symptom reports as input variables (appendix 1 p 12). The transition intensity value represents the rate at which individuals move from one state to another per day given the covariate. The selected covariates are shown in parentheses; the baseline model had the following covariates: Asian, female, age ≥60 years, and weekday (and no symptoms reported, BIL study only). Errors bars represent 95% CIs. BIL=Duke University BIG IDEAs Laboratory.