Reconstructing visual illusory experiences from human brain activity

Abstract

Visual illusions provide valuable insights into the brain's interpretation of the world given sensory inputs. However, the precise manner in which brain activity translates into illusory experiences remains largely unknown. Here, we leverage a brain decoding technique combined with deep neural network (DNN) representations to reconstruct illusory percepts as images from brain activity. The reconstruction model was trained on natural images to establish a link between brain activity and perceptual features and then tested on two types of illusions: illusory lines and neon color spreading. Reconstructions revealed lines and colors consistent with illusory experiences, which varied across the source visual cortical areas. This framework offers a way to materialize subjective experiences, shedding light on the brain's internal representations of the world.

Fig. 1.. Illusory stimuli and image reconstruction procedure.

(A) Example images of the illusion (left) and control (right) conditions: an illusory line induced by offset-gratings (top), the Ehrenstein (center), and Varin (bottom) configurations for neon color spreading. (B) Training. The stimulus features of natural images were extracted with a DNN pretrained for object recognition. Decoders were trained to predict the stimulus DNN features from fMRI responses to the same images. (C) Testing. Illusory images were presented together with control and positive control images. The stimulus features of a test stimulus or the DNN features decoded from fMRI responses to the test stimulus were passed to a pretrained generator for reconstruction.

Fig. 2.. Reconstructions of illusory and control images.

Reconstructions from stimulus features and from brain-decoded features are shown for two representative subjects (S1 and S2). Reconstructions from brain-decoded features were produced from single-trial (8-s) fMRI signals in the whole visual cortex (VC). Representative reconstructions from four different trials are shown for each subject.

Fig. 3.. Evaluation of line illusion reconstructions.

(A) Principal orientation detection. The orientation with the largest variance in Radon projections across line positions was identified as the principal orientation in an image. (B) Distribution of principal orientations in single-trial reconstructions from visual contex (VC) (results for seven subjects and all 90°-difference configurations are pooled, totaling n samples; bin size = 15°). An illusory orientation (star) and an inducer orientation (square) are shown for reference. (C) Comparison of reconstructions from brain-decoded features of VC and stimulus features with added noise. (D) Comparison of reconstructions with the different numbers of inducer lines. (E) Local presence of illusory orientation in reconstructions. (F) Comparison of reconstructions from individual visual areas. Reconstruction examples are from single-trial brain activity in VC [except in (F)] of subject S2. The polar plots show the distributions of the principal orientations pooled across all subjects and all 90°-difference configurations. The bar graphs indicate the proportions of principal orientations closer to the illusory than to the inducer orientation, pooled for all subjects and configurations. Color circles and lines indicate individual subjects. Comparisons with a statistically significant difference at the individual level are marked by solid circles [(C), (E), and (F)].

Fig. 4.. Evaluation of neon color spreading reconstructions.

(A and B) Representative single-trial reconstructions of the illusion (top), control (center), and positive control (bottom) conditions for Ehrenstein from subject S1 (A) and for Varin from subject S2 (B). (C and D) Illustration of regression analysis for comparing the illusion and control conditions for Ehrenstein (C) and Varin (D). The redness map of a reconstructed image was fitted by those of the illusory surface (expected region of color filling-in) and the stimulus. (E and F) Comparison of the illusory surface coefficient values between illusion and control conditions for Ehrenstein (E) and Varin (F). Results for all configurations (sizes and numbers of lines) and seven subjects are pooled for Ehrenstein. Results for six subjects are pooled for Varin. Color lines indicate the results of individual subjects. Comparisons with a statistically significant difference at the individual level are marked by solid circles. (G and H) Illustration of regression analysis for comparing the illusion and the positive control conditions for Ehrenstein (G) and Varin (H). The redness map of a reconstructed image was fitted by that of the illusory or real surface. (I and J) Comparison of the illusory surface coefficient values between the illusion and the positive control conditions for Ehrenstein (I) and Varin (J), pooled as in (E) and (F).