BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity

Abstract

In the visual decoding domain, the most difficult task is the visual reconstruction aimed at reconstructing the presented visual stimuli given the corresponding human brain activity monitored by functional magnetic resonance imaging (fMRI), especially when reconstructing viewed natural images. Recent research regarded the visual reconstruction as the conditional image generation on fMRI voxels and started to use the generative adversarial networks (GANs) to design computational models for this task. Despite the great improvement in previous GAN-based methods, the fidelity and naturalness of the reconstructed images are still unsatisfactory, the reasons include the small number of fMRI data samples and the instability of GAN training. In this study, we propose a new GAN-based Bayesian visual reconstruction model (GAN-BVRM) to avoid the contradiction between naturalness and fidelity in current GAN-based methods. GAN-BVRM is composed of a classifier to decode the categories from fMRI data, a pre-trained conditional generator of the distinguished BigGAN to generate natural images of the specified categories, and a set of encoding models and an evaluator to evaluate the generated images. Composed of neural networks, GAN-BVRM is fully differentiable and can directly generate the reconstructed images by iteratively updating the noise input vector through backpropagation to fit the fMRI voxels. In this process, the decoded categories and encoding models are responsible for the semantic and detailed contents of the reconstructed images, respectively. Experimental results revealed that GAN-BVRM improved the fidelity and naturalness, which validated the advantage of the combining of GANs and Bayesian manner for visual reconstruction.

Keywords: Bayesian framework; brain decoding; generative adversarial network (GAN); visual reconstruction.