Tools of the Trade Multivoxel pattern analysis in fMRI: a practical introduction for social and affective neuroscientists

Abstract

The family of neuroimaging analytical techniques known as multivoxel pattern analysis (MVPA) has dramatically increased in popularity over the past decade, particularly in social and affective neuroscience research using functional magnetic resonance imaging (fMRI). MVPA examines patterns of neural responses, rather than analyzing single voxel- or region-based values, as is customary in conventional univariate analyses. Here, we provide a practical introduction to MVPA and its most popular variants (namely, representational similarity analysis (RSA) and decoding analyses, such as classification using machine learning) for social and affective neuroscientists of all levels, particularly those new to such methods. We discuss how MVPA differs from traditional mass-univariate analyses, the benefits MVPA offers to social neuroscientists, experimental design and analysis considerations, step-by-step instructions for how to implement specific analyses in one's own dataset and issues that are currently facing research using MVPA methods.

Keywords: classification; fMRI; multivoxel pattern analysis; representational similarity analysis; social neuroscience.

Fig. 1

Comparing data in univariate analyses and MVPA. This figure illustrates the differences between how data elicited by four stimuli or experimental conditions (i.e. viewing young and old human and dog faces) is used in univariate analyses (A, B) and MVPA (C, D) as well as how to test a region defined a priori (A, C) vs at every point in the brain (B, D). Each method results in data for each condition (right) that is analyzed and compared (see Figures 3 and 4). (A) Univariate analyses in regions defined a priori use a summary statistic (e.g. mean or peak value) to describe the response magnitude across the entire region. (B) Univariate analyses may also be performed on every voxel independently (mass-univariate analyses). (C) MVPA in regions defined a priori use the pattern of neural responses across all voxels strung out into a vector. (D) In a searchlight analysis, a sphere (here with a radius of two voxels) is defined around every voxel, and the pattern of responses in this sphere is strung out in a vector for each condition. The resulting values displayed on the right are then used in the analyses described in other figures.

Fig. 2

First- and second-order isomorphisms. (A) Neural data from two participants (sets of blue and green squares represent response patterns across voxels) and behavioral ratings (gray) for three stimuli: a happy human face, a sad human face and a giraffe face. (B) Testing for a first-order isomorphism involves directly comparing neural response patterns across people or to behavioral ratings (e.g. testing whether a happy face elicits the same pattern or magnitude of neural responses across participants or whether stimuli that elicit a higher rating also elicit greater neural responses). (C) RDMs capture how, and to what extent, a measure (e.g. responses in a particular brain region, behavioral ratings) distinguishes between stimuli. (D) Testing for a second-order isomorphism involves comparing the relations among stimuli (i.e. comparing RDMs). Here, we see that there are no direct correspondences (i.e. first-order isomorphisms) between neural responses across people or between neural responses and behavioral ratings (B), but there are second-order isomorphisms (D). In other words, even though participants 1 and 2 show different neural response patterns, the highlighted brain region in both participants treats the two human faces as similar to one another, but distinct from a giraffe face. In the same way, even though the amount of activity in this brain region does not directly correspond with behavioral ratings (B), the behavioral data show that the two human faces are identical to each other, and quite distinct from the giraffe face along the rated dimension, mirroring the neural dissimilarity structures. RSA tests for second-order isomorphisms, thus facilitating comparisons across people, modalities of measurement and models, when direct correspondences are impractical or impossible to establish.

Fig. 3

Classification analysis. (A) Within each participant, an algorithm is trained on a subset (here, 9 out of 10 runs) of a participant's data and then tested on a previously unseen subset (here, the heldout run). In the training phase, each sample (here, themultivoxel pattern for each condition in each run) is treated as a point in a representational space. Formvoxels per sample, there aremdimensions in the representational space. Each sample's coordinates are defined by the magnitude of each voxel's response (i.e. voxel 1's response magnitude = coordinate along axis 1, etc.). In many commonly used classification methods, the algorithm then tries to define a boundary (in linear SVM learning, a (m-1)-dimensional hyperplane) in the space such that each sample is classified with its correct label (note that the illustration is merely a conceptual example; please see the main text for a more specific discussion of how particular classification algorithms work). (B) After calibrating model parameters on the training data, the algorithm is then fed the testing data, which it has never seen, without the correct labels. Depending on where those samples fall in the representational space, the algorithm classifies them based on the distinctions it has learned from the training data. If a sample was incorrectly classified, it is counted as an error. (C) The average accuracy across all data folds is calculated for each participant. (D) Repeat this process for each participant, and compare the group-level accuracy to what would be expected based on random chance.

Fig. 4

Representational similarity analysis. Representational similarity analysis (RSA) can be used to create (and, often, compare) RDMs summarizing (A) neural, (C) behavioral and (E) model-based data. (A) To create the neural RDM, the patterns of neural responses elicited by each condition within a particular region are compared with each other to estimate their relative distinctiveness (e.g. the correlation distance between them, 1-r). These distances are organized into a neural RDM. Since the RDM is symmetric about a diagonal of zeros, only the lower off-diagonal triangle of this matrix is extracted, which can be (B) visualized in a low-dimensional space using MDS or (D) compared to a behavioral dissimilarity structure. (B) The MDS plot visualizes the dissimilarity structure by plotting conditions that are more similar closer together. Here, we can see that human faces cluster together and are separate (i.e. dissimilar) from dog faces.We can also see that there seems to be an effect of age, such that young faces are similar to each other and separate from older faces. (C) This effect of perceived age can be tested by creating a behavioral dissimilarity structure. This is achieved by finding the absolute difference between the perceived youth of each pair of faces. Again, the lower off-diagonal triangle is extracted. (D) The lower off-diagonal triangles of the neural and behavioral RDMs are compared with one another, often using the Spearman correlation, as it does not assume a linear mapping between RDMs. This correlation coefficient is mapped back into the region, creating a map of how closely the neural data matches the behavioral ratings. (E) A model RDM of species reflects if two pictures are of the same species or not. (F) Multiple RDMs can be included as predictors in a regression, and the resulting betas may be mapped back into the ROI as an indicator of how much that variable predicted the neural data over and above the other predictor(s).

Fig. 5

Nested k-fold cross-validation with hyperparameter tuning. Cross-validation consists of iteratively splitting data into training and testing datasets, training an algorithm on the training data and then testing the resulting model on the testing data. For each of the k divisions of the data (i.e. folds), hyperparameter tuning may be performed within the training data for that fold. To perform hyperparameter tuning, one would further split the training data into a number of 'sub-folds' consisting of sub-training and validation datasets. Within each of these 'sub-folds', the algorithm is trained on the sub-training data and tested on the validation data once per hyperparameter set. Once every unique combination of hyperparameters has been tested in every 'sub-fold', the hyperparameter set with the best performance across the validation datasets (within the training data) is selected. The selected hyperparameter set is then used to train the algorithm on the entire set of training data for that fold. The resulting model is then tested on the testing data in that fold. This process is repeated for each fold (i.e. k times). Finally, the average performance of the algorithm across all testing datasets is calculated.