Unleashing the full potential of digital outcome measures in clinical trials: eight questions that need attention

Abstract

The use of digital health technologies to measure outcomes in clinical trials opens new opportunities as well as methodological challenges. Digital outcome measures may provide more sensitive and higher-frequency measurements but pose vital statistical challenges around how such outcomes should be defined and validated and how trials incorporating digital outcome measures should be designed and analysed. This article presents eight methodological questions, exploring issues such as the length of measurement period, choice of summary statistic and definition and handling of missing data as well as the potential for new estimands and new analyses to leverage the time series data from digital devices. The impact of key issues highlighted by the eight questions on a primary analysis of a trial are illustrated through a simulation study based on the 2019 Bellerophon INOPulse trial which had time spent in MVPA as a digital outcome measure. These eight questions present broad areas where methodological guidance is needed to enable wider uptake of digital outcome measures in trials.

Keywords: Clinical trial; Digital endpoints; Digital health technology; Digital outcome measures; Validation.

Fig. 1

Eight questions for digital outcome measures: Q1–Q4 relate to validation, Q5 relates to early phase trials, and Q5–Q8 relate to late phase trials

Fig. 2

V3+ framework from the Digital Medicine Society. Original source: [33], reprinted with permission

Fig. 3

Seasonality. Plots show estimated mean of treatment effect (top) and its standard error (bottom) without interaction between season and treatment (left) and with an interaction (right). The seasonal effect varies between 0 and 10. Error bars indicate $1.96\times$ Monte Carlo error. The black error bars indicates the scenario under no seasonal effect. Dark blue, purple and teal lines indicate that the proportion of patients recruited in winter are 0.1, 0.2 and 0.5, respectively. Results are based on 10,000 simulations

Fig. 4

Observer effect and measurement period. Plots show the estimated mean of treatment effect (top), standard error when the measurement period is 4 weeks (middle) and standard error when the measurement period is 2 weeks (bottom), without an interaction between the observer effect and treatment (left) and with an interaction (right). Note that the scale of the y-axis is different for the middle and bottom panels. The observer effect varies between 0 and 10. Error bars indicate $1.96\times$ Monte Carlo error. Grey lines indicate when the measurement period is four weeks, and purple lines indicate when the measurement period is 2 weeks. Results are based on 10,000 simulations

Fig. 5

Missing data. Plots show the change in the estimated mean of treatment effect (top) and its standard error (bottom) when data are MCAR (left) and MNAR (right). The proportion of days that are missing completely at random varies between 0.05 and 0.5. Error bars indicate $1.96\times$ Monte Carlo error. The black error bar indicates the scenario under complete data. Dark green, purple and light green lines indicate that the proportion of patients with missing data are 0.1, 0.2 and 0.5, respectively. Note that the scale of the y-axis is different for the left and right panels for standard error. Results are based on 10,000 simulations

Fig. 6

Daily summaries of MIMS triaxial are displayed for one participant. The mean MIMS triaxial value for the week is 10480.66