Measuring self-regulation in everyday life: Reliability and validity of smartphone-based experiments in alcohol use disorder

Abstract

Self-regulation, the ability to guide behavior according to one's goals, plays an integral role in understanding loss of control over unwanted behaviors, for example in alcohol use disorder (AUD). Yet, experimental tasks that measure processes underlying self-regulation are not easy to deploy in contexts where such behaviors usually occur, namely outside the laboratory, and in clinical populations such as people with AUD. Moreover, lab-based tasks have been criticized for poor test-retest reliability and lack of construct validity. Smartphones can be used to deploy tasks in the field, but often require shorter versions of tasks, which may further decrease reliability. Here, we show that combining smartphone-based tasks with joint hierarchical modeling of longitudinal data can overcome at least some of these shortcomings. We test four short smartphone-based tasks outside the laboratory in a large sample (N = 488) of participants with AUD. Although task measures indeed have low reliability when data are analyzed traditionally by modeling each session separately, joint modeling of longitudinal data increases reliability to good and oftentimes excellent levels. We next test the measures' construct validity and show that extracted latent factors are indeed in line with theoretical accounts of cognitive control and decision-making. Finally, we demonstrate that a resulting cognitive control factor relates to a real-life measure of drinking behavior and yields stronger correlations than single measures based on traditional analyses. Our findings demonstrate how short, smartphone-based task measures, when analyzed with joint hierarchical modeling and latent factor analysis, can overcome frequently reported shortcomings of experimental tasks.

Keywords: Behavioral tasks; Information sampling; Reliability; Risk-taking; Smartphone; Stop signal task; Validity; Working memory.

Fig. 1

Illustration of study timeline and smartphone-based tasks. After an in-lab session for study inclusion and app installation, participants performed four tasks (twice in random order) from a customized version of the Great Brain Experiment (GBE, translated to German) at home. This included a response inhibition, a working memory, a risk-taking, and an information sampling task

Fig. 2

Test–retest reliabilities (ICC1s) for the four tasks split by separate (blue) vs. joint approach (red). Panels show reliabilities for the main outcomes of the experimental tasks. A Stop signal reaction times (SSRTs) for the inhibition task. B Four task outcomes of the working memory task, consisting of no distractor with long encoding time (no [long]), no distractor with short encoding time (no [short]), distractor cues presented at the same time as the patterns (encoding), and distractors presented after the patterns (delayed). C Three main outcomes of the risk-taking task, consisting of risk-taking in a gain context (win), risk-taking in a loss context (loss), and risk taking when gains and losses are mixed (mixed). D Main outcome of the information sampling task, the degree of sampling bias

Fig. 3

A Scree plot used to determine the number of factors best representing the data. B Factor loadings for each of the three extracted factors and each of the tasks’ main outcome variables (for explanations of the task outcomes, see Fig. 2). Factor loadings can range from – 1 to 1, where 1 indicates that a variable is fully described by a factor, 0 that there is no relationship between the factor and the variable, and -1 indicates that the variable is fully described by the inverse of the factor. On top of the factor loadings, Panel B shows the hierarchical tree diagram generated by the clustering analysis.