Deconstructing multivariate decoding for the study of brain function

Abstract

Multivariate decoding methods were developed originally as tools to enable accurate predictions in real-world applications. The realization that these methods can also be employed to study brain function has led to their widespread adoption in the neurosciences. However, prior to the rise of multivariate decoding, the study of brain function was firmly embedded in a statistical philosophy grounded on univariate methods of data analysis. In this way, multivariate decoding for brain interpretation grew out of two established frameworks: multivariate decoding for predictions in real-world applications, and classical univariate analysis based on the study and interpretation of brain activation. We argue that this led to two confusions, one reflecting a mixture of multivariate decoding for prediction or interpretation, and the other a mixture of the conceptual and statistical philosophies underlying multivariate decoding and classical univariate analysis. Here we attempt to systematically disambiguate multivariate decoding for the study of brain function from the frameworks it grew out of. After elaborating these confusions and their consequences, we describe six, often unappreciated, differences between classical univariate analysis and multivariate decoding. We then focus on how the common interpretation of what is signal and noise changes in multivariate decoding. Finally, we use four examples to illustrate where these confusions may impact the interpretation of neuroimaging data. We conclude with a discussion of potential strategies to help resolve these confusions in interpreting multivariate decoding results, including the potential departure from multivariate decoding methods for the study of brain function.

Keywords: Decoding; Encoding; Multivariate analysis; Multivariate decoding; Multivariate pattern analysis; Prediction; fMRI.

Figure 1

The two sources of confusion in multivariate decoding. A. Multivariate decoding was developed for predictions in real-world applications, but is widely used for interpretations about brain function. Since both approaches are often treated as a unitary method despite making different assumptions, this provides a source for confusion. B. The choice between classical univariate analysis is not only a choice of method but a choice of underlying philosophy, activation-based or information-based. Confusion can arise when the conceptual and statistical framework underlying classical univariate analysis is applied to multivariate decoding.

Figure 2

Six differences between classical univariate analysis and multivariate decoding.

Figure 3

The prevailing view of signal and noise in neuroimaging, and its correspondence to information content in multivariate decoding. A. Motivated by the activation-based philosophy, signal reflects the multivariate means of the data, while noise can be either condition-independent error (variance, covariance, or non-normal error), or condition-dependent error (heteroskedastic variance or covariance, or confounds correlating with the conditions). B. Three examples comparing the correspondence of signal and signal-to noise with the weights and accuracy of a linear classifier. In Example 1, the classifier weights reflect the signal, and the accuracy mirrors the signal-to-noise ratio. In Example 2, noise covariance picked up by the classifier causes a departure from this correspondence. In Example 3, despite the absence of signal, differences in noise distribution allow above chance classification, leading to a non-correspondence of the signal to the classifier weights and accuracy.

Figure 4

The accuracy of a classifier is not a standardized estimate of effect size, because it depends on choices such as averaging or the cross-validation scheme. For example, classification accuracies will be lower for single image decoding, but will increase when data within each class are averaged together. However, this need not translate to increased statistical power, because the accuracy estimate is based on fewer responses, increasing their variability. The confusion likely arises from the view that high decoding accuracies are necessary for a decoding model to be useful, which is often true in multivariate decoding for prediction but not multivariate decoding for interpretation.

Figure 5

Different interpretations of “removal of univariate response” from multivariate pattern. A. Original patterns. The response pattern is different across the two conditions. B. Removal of all univariate response differences. This approach removes any univariate differences between conditions from every voxel individually, leaving only the variability across trials. C. Removal of mean response. For each condition, the “overall activation difference” across voxels in a pattern is estimated and then removed from the response pattern. D. Removal of common pattern. The mean response pattern across both conditions is calculated and in another step scaled to optimally fit each individual response pattern. What remains as the corrected pattern is the (collinear) residuals of this fit.

Figure 6

Effect of noise on cross-classification accuracies. A. Differences in the variability of data can affect cross-classification accuracies, despite there being the same effect in the difference of the multivariate means. However, a classifier trained on the noisy dataset would not perform well, either. B. Differences in the covariance of data can affect cross-classification accuracies, even when the general noise level does not vary. Here a classifier trained on the second dataset would perform equally, showing no asymmetries in classification or cross-classification.

Figure 7

How differences in estimability between conditions can contribute to decodability despite an absence of differences in the data. The beta weights for Condition A can be estimated quite well, because this regressor is largely orthogonal to the other regressors, while the regressor for Condition B is non-orthogonal to the regressor of Condition C. As a consequence, on average, both beta estimates will be close to the true value. However, since the regressor for Condition B is non-orthogonal with Condition C, the estimation will be more variable. Classical methods would not reveal any differences between conditions. In contrast, as has been illustrated in Figure 3C, a multivariate classifier can pick up this difference in variability, which can lead to above-chance decoding accuracies even in the absence of any difference in the data. The reason for this discrepancy lies in the different meaning of signal and noise in the standard statistical framework and the information-based framework.