Improving the Reliability of Computational Analyses: Model-Based Planning and Its Relationship With Compulsivity

Abstract

Background: Computational models show great promise in mapping latent decision-making processes onto dissociable neural substrates and clinical phenotypes. One prominent example in reinforcement learning is model-based planning, which specifically relates to transdiagnostic compulsivity. However, the reliability of computational model-derived measures such as model-based planning is unclear. Establishing reliability is necessary to ensure that such models measure stable, traitlike processes, as assumed in computational psychiatry. Although analysis approaches affect validity of reinforcement learning models and reliability of other task-based measures, their effect on reliability of reinforcement learning models of empirical data has not been systematically studied.

Methods: We first assessed within- and across-session reliability and effects of analysis approaches (model estimation, parameterization, and data cleaning) of measures of model-based planning in patients with compulsive disorders (n = 38). The analysis approaches affecting test-retest reliability were tested in 3 large generalization samples (healthy participants: n = 541 and 111; people with a range of compulsivity: n = 1413).

Results: Analysis approaches greatly influenced reliability: reliability of model-based planning measures ranged from 0 (no concordance) to above 0.9 (acceptable for clinical applications). The largest influence on reliability was whether model-estimation approaches were robust and accounted for the hierarchical structure of estimated parameters. Improvements in reliability generalized to other datasets and greatly reduced the sample size needed to find a relationship between model-based planning and compulsivity in an independent dataset.

Conclusions: These results indicate that computational psychiatry measures such as model-based planning can reliably measure latent decision-making processes, but when doing so must assess the ability of methods to estimate complex models from limited data.

Keywords: Compulsivity; Computational modeling; Computational psychiatry; Psychometrics; Reinforcement learning; Reliability.

Figure 1.. Task schematic.

Each trial of the task consisted of two stages. At the first stage, participants chose between two stimuli (spaceships), which probabilistically led to one (70%, thick line) or the other (30%, dotted line) second stage set of stimuli (aliens on planets). The second stage stimuli led to reward based on a probability associated with each second stage stimulus (red vs. purple for aliens from Planet 1 vs. Planet 2; solid vs. dashed lines for the Left vs. Right alien on each planet). This probability drifts independently throughout the task for each stimulus, requiring continual learning. The effect of model-based planning is illustrated by behavior after a trial with a rare first-to-second-stage transition that results in reward: with high model-based planning, a participant using her internal model of the task, including knowledge of the transition probabilities, would realize that she needs to switch first-stage options to maximize the probability of entering the same second-stage state and receiving reward again. Meanwhile with low model-based planning, she would only use her direct experience of choosing an option and receiving a reward, and disregarding transition probabilities, she would stay with the same first stage option that had previously led to reward.

Figure 2.. Test-retest reliability of model-based measures.

Reliability values are shown overall and grouped together by model estimation, model parameterization, and data cleaning approaches. Dots indicate reliability of individual combinations of approaches and thick horizontal lines the median for each approach. Parameters and regression coefficients measuring model-based learning were: reward x transition interaction for logistic regression, ω (model-based versus model-free weight) for M1, and β1 MB (model-based beta, signifying the influence of first stage model-based values on choices) for M2 and M3. Reliability values for each combination of approaches are reported in Supplementary Table 2.

Figure 3.. Split-half reliability of model-based measures.

Reliability values of odd compared to even trials from session 1 are shown overall and grouped together by model estimation, model parameterization, and data cleaning approaches. Dots indicate reliability of individual combinations of approaches and thick horizontal lines the median for each approach. Parameters and regression coefficients measuring model-based learning were: reward x transition interaction for logistic regression, ω (model-based versus model-free weight) for M1, and β1 MB (model-based beta, signifying the influence of first stage model-based values on choices) for M2 and M3. Reliability values for each combination of approaches are reported in Supplementary Table 3.

Figure 4.. Generalization of results to test-retest reliability in other datasets and to relationships with clinical variables.

(A) Test-retest reliability of approaches differing in reliability were tested in two independent datasets (S1: 111 participants; S2: 541 participants) and showed similar patterns of reliability. Dots indicate median reliability, colored by estimation method, and lines show 95% confidence or credible intervals (CIs). Reliability values and CIs are reported in Supplementary Table 6. (B) Estimation methods differing in reliability were used to test the relationship between model-based planning and compulsivity in subsets of participants from an independent dataset. X axis is number of participants, randomly drawn from the full sample (n = 1413) and y axis is the reduction in model-based planning with transdiagnostic compulsivity (z-scored). Dotted black horizontal lines indicate significance levels of p = 0.05, p = 0.01, and p=0.001. Colored lines indicate regression lines of the relationship between the number of subjects and the z-scored reduction in model-based planning with compulsivity, colored by estimation approach. Steeper lines reflect more precise estimation in methods with greater reliability. (C) Extrapolated effect sizes (f²) for the effect of compulsivity on model-based planning, extrapolated from the relationship between sample size and significance shown in Figure 4C, with small and medium effect sizes shown for reference.