Computational Phenotyping in Psychiatry: A Worked Example

Abstract

Computational psychiatry is a rapidly emerging field that uses model-based quantities to infer the behavioral and neuronal abnormalities that underlie psychopathology. If successful, this approach promises key insights into (pathological) brain function as well as a more mechanistic and quantitative approach to psychiatric nosology-structuring therapeutic interventions and predicting response and relapse. The basic procedure in computational psychiatry is to build a computational model that formalizes a behavioral or neuronal process. Measured behavioral (or neuronal) responses are then used to infer the model parameters of a single subject or a group of subjects. Here, we provide an illustrative overview over this process, starting from the modeling of choice behavior in a specific task, simulating data, and then inverting that model to estimate group effects. Finally, we illustrate cross-validation to assess whether between-subject variables (e.g., diagnosis) can be recovered successfully. Our worked example uses a simple two-step maze task and a model of choice behavior based on (active) inference and Markov decision processes. The procedural steps and routines we illustrate are not restricted to a specific field of research or particular computational model but can, in principle, be applied in many domains of computational psychiatry.

Keywords: Markov decision process; active inference; computational psychiatry; generative model; model inversion.

Figure 1.

A schematic overview of the analysis stream underlying the treatment of computational psychiatry in this article. The basic procedure involves specifying a model of behavior cast in terms of a Markov decision process (MDP). Under the assumption that choices are made in an approximately Bayes optimal fashion using active (Bayesian) inference, this model is sufficient to predict behavior. If we supplement the model specification (MDP) with empirical choice behavior (Data), we can estimate the prior beliefs responsible for those choices. If this is repeated for a series of subjects, the ensuing priors can then be analyzed in a random-effects (PEB) model to make inferences about group effects or to perform cross-validation. Furthermore, physiological and behavioral predictions can be used as expansion variables for fMRI or other neuroimaging time series (bottom left). The routines in the boxes refer to MATLAB routines that are available in the academic software SPM. These routines are sufficient to both simulate behavioral responses and analyze empirical or observed choice behaviour, at both the within-subject and between-subject levels. The final routine also enables cross-validation and predictions about a new subject’s prior beliefs using a leave-one-out scheme that may be useful for establishing the predictive validity of any models that are considered.

Figure 2.

A, Task: we used a simple two-step maze task for our simulations, where a subject starts in the middle of a T-shaped maze and has to decide whether to sample the left or right arm, knowing that one of the two arms will contain a reward but it can sample only one of them (i.e., the arms are absorbing states). Alternatively, the subject could sample a cue at the bottom of the maze, which will tell her which arm to sample. B, State space: Here, the subject has four different control states or actions available: she can move to the middle location, the left or the right arm or the cue location. Based on these control states, we can specify the hidden states, which are all possible states that a subject can visit in a task and often are only partially observable. In this task, the hidden state comprises the location × the context (reward left or right), resulting in 4 × 2 = 8 different hidden states. Finally, we have to specify the possible outcomes or observations that an agent can make. Here, the subject can find itself in the middle location, in the left or right arm with or without obtaining a reward or at the cue location.

Figure 3.

Generative model. A, The A-matrix maps from hidden states to observable outcome states (resulting in a 7 × 8 matrix). There is a deterministic mapping when the subject is either in the middle position (simply observing that she is in the middle) or at the cue location (simply observing that she is at the cue location where the cue indicates either left or right). However, when the subject is in the left or right arm, there is a probabilistic mapping to a rewarded and an unrewarded outcome. For example, if the subject is in the left arm and the cue indicated is in the left arm (third column), there is a high probability, p, of a reward, whereas there is a low probability q = 1 − p of no reward. B, The B-matrix encodes the transition probabilities (i.e. the mapping from the current hidden state to the next hidden state contingent on the action taken by the agent). Thus, we need as many B-matrices as there are actions available (four in this example). Illustrated here is the B-matrix for a move to the left arm. We see that the action never changes the context, but (deterministically) does change the location, by always bringing it to the left arm, except when starting from an absorbing state (right arm). C, Finally, we have to specify the preferences over outcome states in a C-vector. Here, the subject strongly prefers ending up in a reward state and strongly dislikes ending up in a left or right arm with no reward, whereas it is somewhat indifferent about the “intermediate” states. Note that these preferences are (prior) beliefs or expectations; for example, the agent beliefs that a rewarding state is $\mathrm{e}\mathrm{x}\mathrm{p}\left(4\right)\approx 55$ times more likely than an “intermediate” state [exp(0) = 1].

Figure 4.

Data simulation using the routine spm_MDP_VB. A, A simulated example trial, where the left top panel shows the hidden states, the right top panel shows the inferred actions, the middle panels show the inference on policies (i.e., the possible sequences of actions), the bottom left panel shows the preferences over outcome states (c-vector), and the bottom right panel shows the expected precision, which could be encoded by dopamine (Friston et al., n..). In this trial, the subject starts in the middle position where the reward is (most likely) on the right arm. She then makes a selection to sample the cue and, finally, moves to the right arm, as indicated by the cue (darker colors reflect higher posterior probabilities). B, Overview of a simulated experiment comprising 128 trials. The first panel shows the inferred policies (black regions) and initial states (shown as colored circles: red circles, reward is located at the right arm; blue circles, reward is located at the left arm) at every given trial. The second panel shows estimated reaction times (cyan dots), outcome states (colored circles), and the value of those outcomes (black bars). Note that the value of outcomes is expressed in terms of an agent’s (expected) utility, which is defined as the logarithm of an agent’s prior expectations. Thus, the utility of an outcome is at most 0 [= log(1)]. Reaction times reflect the choice conflict at any given trial and are simulated by using the time it takes Matlab to simulate inference and subsequent choice in any given trial (using the tic-toc function in Matlab). The third and fourth panels show simulated event-related potentials for hidden state estimation and expected precision, respectively. The specifics of these simulations are discussed in detail elsewhere (Friston et al., 2016). Finally, panels five and six illustrate learning and habit formation. Our simulations did not include any learning or habitual responses.

Figure 5.

Model inversion, as implemented by the routine spm_dcm_mdp, is based on simulated behavior. In this routine, (negative) variational free energy as a lower bound of log-model evidence is maximized and converges after the 13th iteration (top right). The trajectory of two estimated parameters in parameter space is provided (top left) as well as their final conditional estimates (bottom left) and their posterior deviation from the prior value (bottom right). The black bars on the bottom right show the true values, while the gray bars show the conditional estimates, illustrating a characteristic shrinkage toward the prior mean.

Figure 6.

A, Conditional estimate and confidence interval for the hyperprior on precision (β) as a function of the number of trials in a simulated experiment. B, True and estimated subject-specific parameters, following model inversion for 16 subjects with a group effect in the hyperprior (β). The two groups can be seen as two clusters along the diagonal.

Figure 7.

Hierarchical empirical Bayesian inference on group effects using the function spm_dcm_peb. A, Results of Bayesian model comparison (reduction) to infer whether the full model (with both group mean and group differences) or a reduced (nested) model (bottom left) provides a better explanation for the data. These results indicate high posterior evidence for a model with a group difference, with slightly less evidence for the full model, which also includes a group mean effect (i.e., a deviation from the group prior mean; top panels). Middle panels show the maximum a posteriori estimates of the mean and group effects for the full and reduced models. B, Estimated (gray bars) group mean (left) and difference (right) in β. These estimates are about one-quarter (top right), which corresponds to the group effect that was introduced in the simulations (black bars). The small bars correspond to 90% Bayesian confidence intervals. A reduced parameter estimate corresponds to the Bayesian model average over all possible models (full and reduced) following Bayesian model reduction.

Figure 8.

Cross-validation based on a leave-one-out scheme. Using the function spm_dcm_loo, we find that group membership is accurately recovered based on the parameter estimate of the hyperprior on each subject. This is evidenced by a high correlation between inferred and true group membership in the top right panel. These reflect out-of-sample estimates of effect sizes, which were large (by design) in this example. The top right panel provides the estimate of the group indicator variable (which is +1 for the first group and −1 for the second group). The bottom panel provides the posterior probability that each subject belongs to the first group.

Figure 9.

Simulated group difference between control subjects and patients (with a group difference in precision of one-quarter) in the average reward received. Note that this difference in an observable variable was successfully traced back to a difference in the hyperprior on precision (a latent variable) by our inference scheme, which is important because such inverse problems are usually ill posed and hard to solve.