Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging

Abstract

Human neuroimaging research has transitioned from mapping local effects to developing predictive models of mental events that integrate information distributed across multiple brain systems. Here we review work demonstrating how multivariate predictive models have been utilized to provide quantitative, falsifiable predictions; establish mappings between brain and mind with larger effects than traditional approaches; and help explain how the brain represents mental constructs and processes. Although there is increasing progress toward the first two of these goals, models are only beginning to address the latter objective. By explicitly identifying gaps in knowledge, research programs can move deliberately and programmatically toward the goal of identifying brain representations underlying mental states and processes.

Keywords: affect; brain signature; decoding; fMRI; machine learning; multivariate; pain; pattern recognition; population coding; representation.

Figure 1.

Brain maps versus brain models. (A) The objective of conventional brain mapping is to identify which brain regions are reliably more active as a function of different kinds of stimulation or manipulations of mental state (in addition to error, ɛ). The classical outcome of brain mapping study is a parametric map indicating the extent to which every brain measure (voxel) is associated with a given mental state. The objective of developing a multivariate brain model is to account for, and thus predict, an individual person’s mental state or behavior (outcomes) based on their brain activity. (B) Brain maps are displayed for comparisons of brain responses between emotional faces and shapes, reward and punishment (Barch et al., 2013), and painful pressure applied to the thumb and rest (study 5 from Kragel et al., 2018). (C) Brain models can vary in complexity, ranging from the average activity of individual brain regions (e.g., a bilateral amygdala mask [left] (Swartz et al., 2015)) to more complex patterns of brain activity optimized through statistical learning procedures (e.g., the top 1% of voxels that predict crowdfunding choices [center] (Genevsky et al., 2017), or the Neurological Pain Signature [right] (Wager et al., 2013)).

Figure 2.

Advances in multivariate brain modeling. (A) Timeline of methodological developments in predictive brain modeling. Advances in predicting behavioral and mental outcomes are influenced by ideas about local coding, distributed coding, and generalizability. These ideas fostered complementary tools for analyzing brain data. (B) Decisions involved in developing multivariate models of brain activity. Three classes of decisions involve the intended generalizability of a model (should it work for a single individual, or a whole population?), the spatial scale of modeling (should activity within a local searchlight, a single brain system, or the whole brain be modeled?), and the complexity of relationships linking brain and mind (e.g., should a linear or quadratic function be used to map brain activity to model outcomes?). The right most column depicts multivariate brain models that are the result of different methodological decisions.

Figure 3.

Examining predictive models at multiple spatial scales. Brain-wide multivariate models can be understood by examining how pattern expression (i.e., model output for test data) varies across established brain networks and regions. (A) Brain-wide expression of the Neurologic Pain Signature (NPS), a signature developed to predict physical pain intensity (Wager et al., 2013), and the Vicarious Pain Signature (VPS), a signature developed to predict observed pain intensity, in comparisons of high versus low levels of heat pain (red) and observed-pain (purple)(data from study 1 of Krishnan et al., 2016). These two signatures are independently affected by these two manipulations. (B) Decomposing a distributed pattern into subsystems: Expression of the NPS and VPS within 7 resting-state networks (Yeo et al., 2011). Wedge plots of the same dataset depict normalized local pattern expression (using the signature weights in the local region), with red indicating positive values and blue negative values. The darker shaded area indicates the standard error of the mean across individuals. The NPS primarily has positive expression during pain in the ‘ventral attention’ and ‘somatomotor’ networks during heat-pain, and negative expression in the ‘dorsal attention’ and ‘limbic’ networks. In contrast, the VPS has more evenly distributed expression across cortical networks, with a peak in the ‘visual network’ during observed-pain. (C). Meso-scale organization: Heat pain and observed pain also have distinct profiles of local pattern responses in the diencephalon, based on an anatomical delineation of thalamic nuclei and hypothalamus into 17 distinct regions (Krauth et al., 2010; Niemann et al., 2000). NPS expression during pain is positive in many thalamic nuclei and negative in the habenula, whereas VPS is expressed most reliably during observed pain in the hypothalamus. Error bars reflect standard error of the mean. (A) adapted from (Krishnan et al., 2016). dAttention = dorsal attention, vAttention = ventral attention, Pulv = pulvinar, LGN = lateral geniculate nucleus, MGN = medial geniculate nucleus, VPL = ventral posterolateral nucleus, VPM = ventral posterior medial nucleus, Intralam = intralaminar nuclei, Midline = midline thalamic nuclei, LD = lateral dorsal nucleus, VL = ventral lateral nucleus, LP = lateral posterior nucleus, VA = ventral anterior nucleus, VM = ventral medial nucleus, MD = mediodorsal nucleus, AM anteromedial nucleus, AV = anteroventral nucleus, Hb = habenular nucleus, Hythal = hypothalamus.

Figure I.

The effect of spatial scale on model performance. The plot shows the average cross-validated performance (Pearson’s r, averaged over 500 iterations) of models designed to predict pain reports following thermal stimulation using a 2-fold subject-independent cross-validation (data from study 2 of Wager et al., 2013; n = 33). The x-axis denotes the number of voxels used in each model, which were sampled randomly from a uniform distribution spanning the entire brain (black) or individual resting-state networks (colored lines, inset render shows each network from a medial view). Solid curves display parametric fits of the form A – Be^−v/C, where v is the number of voxels, A is the performance of the whole-brain model, B is the performance of a single voxel, and C determines the rate of increase. Sampling voxels from the whole brain produces the most predictive models, compared to sampling within a single resting-state network or searchlight, although only ~1,000 randomly sampled voxels are needed to achieve this performance.