Efficient Neural Decoding Based on Multimodal Training

Abstract

Background/objectives: Neural decoding methods are often limited by the performance of brain encoders, which map complex brain signals into a latent representation space of perception information. These brain encoders are constrained by the limited amount of paired brain and stimuli data available for training, making it challenging to learn rich neural representations.

Methods: To address this limitation, we present a novel multimodal training approach using paired image and functional magnetic resonance imaging (fMRI) data to establish a brain masked autoencoder that learns the interactions between images and brain activities. Subsequently, we employ a diffusion model conditioned on brain data to decode realistic images.

Results: Our method achieves high-quality decoding results in semantic contents and low-level visual attributes, outperforming previous methods both qualitatively and quantitatively, while maintaining computational efficiency. Additionally, our method is applied to decode artificial patterns across region of interests (ROIs) to explore their functional properties. We not only validate existing knowledge concerning ROIs but also unveil new insights, such as the synergy between early visual cortex and higher-level scene ROIs, as well as the competition within the higher-level scene ROIs.

Conclusions: These findings provide valuable insights for future directions in the field of neural decoding.

Keywords: diffusion model; fusion transformer; multimodal pre-training; neural decoding; scene reconstruction.

Figure A1

Qualitative results.

Figure 1

Neural decoding of visual image. Subjects undergo fMRI scanning while viewing visual stimuli. The brain activity corresponding to stimuli is recorded and transformed into features. Computational models reconstruct the original stimuli based on the features.

Figure 2

Proposed pipeline of decoding with multimodal training.

Figure 3

Qualitative results.

Figure 4

50-way accuracy and computational complexity.

Figure 5

Images derived from synthetic fMRI patterns generated through the activation of one ROI.

Figure 6

Images derived from synthetic fMRI patterns generated through the activation of two ROIs.