Decoding the visual and subjective contents of the human brain

Abstract

The potential for human neuroimaging to read out the detailed contents of a person's mental state has yet to be fully explored. We investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found that ensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in early visual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and their subjective contents.

Fig. 1

Orientation decoder and ensemble orientation selectivity. (a) The orientation decoder predicts stimulus orientation based on fMRI activity patterns. The cubes depict an input fMRI activity pattern obtained while the subject viewed gratings of a given orientation (left). The circles are “linear ensemble orientation detectors,” each of which linearly combines the fMRI voxel inputs (weighted sum plus bias; the bias component not shown). The weights (W) are determined by a statistical learning algorithm (linear support vector machine) applied to a training data set, such that the output of each detector becomes largest for its “preferred orientation” (θi). The final unit (rectangle with “Max”) decides the prediction to be the preferred orientation of the detector with the highest value. (b) Orientation selectivity of individual voxels and linear ensemble orientation detectors. The decoder was trained using actual fMRI responses to eight orientations (S1, 400 voxels from V1/V2). Average responses are plotted as a function of orientation for two representative voxels, and for 45° and 135° detectors (error bar, standard deviation).

Fig. 2

Decoding stimulus orientation from ensemble fMRI activity in the visual cortex. (a) Decoded orientation responses for eight orientations. Polar plots indicate the distribution of predicted orientations for each of eight orientations (S2, 400 voxels from V1/V2, 22 samples per orientation). The same values are plotted at symmetrical directions as stimulus orientation repeats every 180°. Solid black lines show the true stimulus orientations. (b) Decoded orientation responses for all four subjects (400 voxels from V1/V2, total 160–192 samples for each subject). Results for individual orientations are pooled relative to the correct orientations, and aligned to the vertical line. (c) Across-session generalization. Decoded orientation responses were obtained by training a decoder with Day 1’s data and testing with Day 2’s data (31 days and 40 days apart for S1 and S2, respectively).

Fig. 3

Orientation selectivity across the human visual pathway. Decoded orientation responses are shown for individual visual areas from V1 through V4 and MT+ (S3, 100 voxels per area). The color map indicates t-values associated with the responses to the visual field localizer for V1 through V4, and to the MT+ localizer for MT+ (see Methods). The voxels from both hemispheres were combined to obtain the results, though only the right hemisphere is shown. All other subjects showed similar results of progressively diminishing orientation selectivity in higher areas.

Fig. 4

Pairwise decoding performance as a function of orientation difference (all pairs from eight orientations), for (I) grating images (pixel intensities), (II) fMRI images (voxel intensities), and (III) transformed grating images. The gratings (I) were 20×20 pixel black/white images with 2–3 stripes. The fMRI images (II) were those obtained in the present study (responses to gratings of eight orientations; 400 voxels from V1/V2). The transformed images (III) were created by linear orientation filtering (Gabor-like filters for four orientations) of the grating images (I) followed by thresholding (non-linearity) and addition of noise. The orientations of these images were decoded for each pair of orientations (chance level, 50%). For (I) and (III), the average performance with five sets of phase randomized images is plotted (error bar, standard deviation). For (II), the average performance of four subjects is shown. The grating images (I) resulted in poor performance regardless of orientation difference. In contrast, the fMRI images (II) and the transformed grating images (III) both showed performance that improved with orientation difference, reaching near perfect levels at 90°.

Fig. 5

Orientation preference map on flattened cortical surface. The color maps depict the orientation preference of individual voxels on the flattened surface of left ventral V1 and V2 for subjects S2 and S3 (scale bar, 1 cm). Each cell delineated by thick lines is the cross section of a single voxel (3×3×3 mm) at the gray-white matter boundary. Voxel colors depict the orientation detector for which each voxel provides the largest weight. The overall color map indicates a template pattern that activates each detector most effectively. The weights were calculated using 400 voxels from V1/V2 including all the quadrants. Other subjects also showed scattered but different patterns of orientation preference. Note that the color map indicates only weak preference for one orientation over others. Simple averaging of the voxels with the same orientation preference does not lead to sharp orientation tuning, as found in Fig. 1b (see also Supplementary Fig. 2 online).

Fig. 6

Simulation of one-dimensional array of columns and voxels. Each column was assumed to respond to orientation input according to a Gaussian-tuning function peaking at its preferred orientation (SD, 45°; noise was added to the output). The preferred orientation shifted by a constant degree (a) or by a constant degree plus noise (b). In each trial, a single orientation was given as input, and the outputs of 100,000 columns (color band) were sampled by 100 voxels (gray boxes). The actual location of voxel sampling was randomly jittered on each trial (Gaussian distribution with an SD of a 1/4 voxel size) to take into account residual head motion. The number of stimulation trials was chosen to match the fMRI experiment. The sampled voxel data were analyzed using the same decoding procedure. As can be seen in the polar plots on the right, orientation can be readily decoded from the irregular array of columns (b), but not from the regular array (a). Similar results were obtained with a wide range of simulation parameters. Note that if voxel sampling is no longer jittered to mimic minor brain motion, orientation can be decoded even from the regular column array. This is because the high spatial frequency component can still persist after the sampling by large voxels. However, given that pulsatile brain motion and minor head motion cannot be fully eliminated or corrected with 3D alignment procedures, it seems unlikely that such high-frequency information contributes much to the orientation content in our fMRI data.

Fig. 7

Mind-reading of attended orientation. (a) Mind-reading procedure. First, a decoder was trained using fMRI activity evoked by single gratings to discriminate 45° vs. 135°. Black and white gratings (equal contrast) were counterbalanced across trials. In the attention experiment, a plaid pattern composed of two gratings (black/white counterbalanced) was presented. The color of the central fixation spot was changed to indicate which orientation (45° or 135°) the subject should pay attention to in each trial. The fMRI activity patterns obtained in the attention experiment were classified by the decoder trained with single gratings. (b) Mind-reading performance. Gray lines plot decoded orientation responses for the “attend to 45°” and “attend to 135°” conditions (S3, 800 voxels from V1–V4). Thick lines indicate attended orientations. (c) Mind-reading performance across the human visual pathway. The percentage of correct decoding is plotted by visual area (chance level, 50%; 800 voxels for V1–V4 combined, 200 voxels for V1, V2, V3, and V3a/V4v, and 100 voxels for MT+). Colored lines show the performance of four individual subjects. Black points and lines depict the mean cross-validation performance obtained with single gratings (training session).