Forecasting fMRI images from video sequences: linear model analysis

Abstract

Over the past few decades, a variety of significant scientific breakthroughs have been achieved in the fields of brain encoding and decoding using the functional magnetic resonance imaging (fMRI). Many studies have been conducted on the topic of human brain reaction to visual stimuli. However, the relationship between fMRI images and video sequences viewed by humans remains complex and is often studied using large transformer models. In this paper, we investigate the correlation between videos presented to participants during an experiment and the resulting fMRI images. To achieve this, we propose a method for creating a linear model that predicts changes in fMRI signals based on video sequence images. A linear model is constructed for each individual voxel in the fMRI image, assuming that the image sequence follows a Markov property. Through the comprehensive qualitative experiments, we demonstrate the relationship between the two time series. We hope that our findings contribute to a deeper understanding of the human brain's reaction to external stimuli and provide a basis for future research in this area.

Keywords: Correlation analysis; Forecasting; Hypothesis testing; Linear model; Neuroimaging; Video sequences; fMRI.

Fig. 1

Overview of our proposed forecasting method. Preprocessing stage (left one) involves the image features extraction using the pre-trained image encoder model. fMRI images are preprocessed using normalizing techniques. Forecasting stage (right one) comprises a multi-dimensional linear model, which maps the video frame embedding to the difference between two sequential tensors

Fig. 2

The visualization of data from the utilized sample of subjects under examination. Three points in time were selected for example. The video frames are presented in the left-hand section of the figure. The right-hand side of the diagram shows slices of fMRI scans from the relevant time period for three different subjects: 7, 13 and 47. The video frame rate is greater, therefore each fMRI scan is matched with a bunch of video frames

Fig. 3

Slices of fMRI images from the test sample. The figure presents slices of the true and reconstructed tomographic scan from the test sample and the difference between them. An error on the order of ${10}^{-3}$ indicates a fairly accurate prediction

Fig. 4

Localization of the most active zone. The figure indicates that the localized region is the occipital lobe. This area of the brain is responsible for processing visual information, including information during the viewing of a video sequence, which supports the correctness of the localization

Fig. 5

Dependence of MSE on delay time. On the left side, the graph does not exhibit any point of minimum. We investigate that calculation on the entire fMRI image is noisy. Therefore, we correct the MSE, localizing the occipital lobe, which leads to the distinctive minimum at 5 s (on the right side)

Fig. 6

Dependence of MSE metric on regularization parameter $\alpha$ on images from the test sample. The graphs show that the optimal value of the coefficient is $\alpha \approx 1000$. Increasing the compression ratio reduces quality due to the lower number of background voxels

Fig. 7

Weights vector component distribution. The model weights are not concentrated around any particular value, indicating a non-degenerate distribution. This suggests that the model is well-trained and captures a diverse range of features

Fig. 8

MAPE of MSE changing when predicting on the mixed weight matrix. The figure shows the Mean Absolute Percentage Error (MAPE) of predicted MSE between pairs of subjects. Each row and column represents a subject. "Mixed" MSE uses one subject’s weights to predict another’s outcomes. Positive values indicate "mixed" MSE is greater than the "true" MSE

Fig. 9

Slices of fMRI images from the test sample. Figure shows slices from the ground truth and reconstructed tomographic scan (derived from original video frames using proposed method) along with the differences between them

Fig. 10

Slices of fMRI images from the test sample (uninformative data). The MSE value for the uninformative sample is larger, which indicates the presence of a correlation between the true data