A neural network approach to fMRI binocular visual rivalry task analysis

Abstract

The purpose of this study was to investigate whether artificial neural networks (ANN) are able to decode participants' conscious experience perception from brain activity alone, using complex and ecological stimuli. To reach the aim we conducted pattern recognition data analysis on fMRI data acquired during the execution of a binocular visual rivalry paradigm (BR). Twelve healthy participants were submitted to fMRI during the execution of a binocular non-rivalry (BNR) and a BR paradigm in which two classes of stimuli (faces and houses) were presented. During the binocular rivalry paradigm, behavioral responses related to the switching between consciously perceived stimuli were also collected. First, we used the BNR paradigm as a functional localizer to identify the brain areas involved the processing of the stimuli. Second, we trained the ANN on the BNR fMRI data restricted to these regions of interest. Third, we applied the trained ANN to the BR data as a 'brain reading' tool to discriminate the pattern of neural activity between the two stimuli. Fourth, we verified the consistency of the ANN outputs with the collected behavioral indicators of which stimulus was consciously perceived by the participants. Our main results showed that the trained ANN was able to generalize across the two different tasks (i.e. BNR and BR) and to identify with high accuracy the cognitive state of the participants (i.e. which stimulus was consciously perceived) during the BR condition. The behavioral response, employed as control parameter, was compared with the network output and a statistically significant percentage of correspondences (p-value <0.05) were obtained for all subjects. In conclusion the present study provides a method based on multivariate pattern analysis to investigate the neural basis of visual consciousness during the BR phenomenon when behavioral indicators lack or are inconsistent, like in disorders of consciousness or sedated patients.

Figure 1. Diagrams showing the fMRI block tasks design.

The letter H in the red box represents the house block, the letter F in the blue box represents the face block, and the white cross in the black box represents the rest block. The BNR task is shown on the top, while the BR task is shown on the bottom.

Figure 2. Diagram illustrating signal processing steps.

In the green boxes the steps concerning the BNR ROI signals are described, in the blue boxes the steps concerning BR ROI signals are described, and in the orange boxes the steps in common for both signals are described.

Figure 3. Example of resulting BOLD activity from GLM single-subject analysis of BNR-localizer task.

Picture A shows t-contrast activations of face-house (FWE<0.05) in BNR time-course of PPA on top and FFA ROI on the bottom. Picture B shows t-contrast activations of house-face (FWE<0.05) in BNR-localizer time-course of FFA on top and PPA ROI on the bottom.

Figure 4. Bar plot of ANN percentage of successes for each subject.

The plot shows in ordinate the percentage of time points correctly allocate to conditions (house or face) and in abscissa the number associated to the participants.

Figure 5. Examples of stimuli distributions after 1000 output repetitions.

In panel A on top the bar histogram shows for subject 7 the results of the 1000 reiterations using the BR task time course signal. In y axis is represented the percentage number (frequency) of house and face over all time points in a reiteration, and in the x axis the percentage number of reiterations (events) in which we obtained a determined frequency of houses and faces over all 1000 reiterations. In the plot on the bottom of panel A the BR time course signal has been replaced with Rest time course signal. In panel B the same bar histograms, for BR and rest time courses, are shown for the subject excluded because he did not experience the BR phenomenon.