Real-time decoding of covert attention in higher-order visual areas

Abstract

Brain-computer-interfaces (BCI) provide a means of using human brain activations to control devices for communication. Until now this has only been demonstrated in primary motor and sensory brain regions, using surgical implants or non-invasive neuroimaging techniques. Here, we provide proof-of-principle for the use of higher-order brain regions involved in complex cognitive processes such as attention. Using realtime fMRI, we implemented an online 'winner-takes-all approach' with quadrant-specific parameter estimates, to achieve single-block classification of brain activations. These were linked to the covert allocation of attention to real-world images presented at 4-quadrant locations. Accuracies in three target regions were significantly above chance, with individual decoding accuracies reaching upto 70%. By utilising higher order mental processes, 'cognitive BCIs' access varied and therefore more versatile information, potentially providing a platform for communication in patients who are unable to speak or move due to brain injury.

Fig. 1

Graphs showing the modeled brain responses to m-sequences by convolving the HRF with the delta functions for the m-sequence for each quadrant. (a) Timeseries for each quadrant, showing the relative orthogonality for each quadrant. (b) Degree of correlation between the timeseries from the ‘localiser’ session with the ‘attended’ quadrant (red line) versus the other three simultaneous presented quadrant-based stimulus streams. By introducing a weighting to each of the quadrant time series, we examined if it would make it more discrete from the other three. The introduction of weighting served to mimic the effect of attention.

Fig. 2

‘Cued attention’ session schematic. Participants were cued to attend stimuli presented in one quadrant per block. The directional cue stimulus was a stick man pointing towards the quadrant to be attended (first screen). During stimulus presentation in the 4 quadrants (i.e. second screen), blank stimuli (shown as black images) were interspersed with stimuli from the other four categories (i.e. faces, houses, body parts, food/drink), enabling quadrant-specific m-sequences to be used for stimulus presentation. During the rest block (i.e. third screen) participants maintained central eye fixation, facilitated by a white dot at the centre of the screen.

Fig. 3

‘Non-cued’ sessions schematic. Participants were instructed to fixate centrally, and attend to one of four quadrants stimulus presentations for the duration of the block. They disclosed which quadrant they had attended at the end of each block using a button-box. Stimuli included four categories (faces, houses, household objects, body parts). ‘Blank’ stimuli (represented by black icons) appeared in a quadrant-specific fashion in keeping with a quadrant-specific m-sequence.

Fig. 4

Participant-averaged decoding accuracy for the three ROIs averaged across sessions and blocks. Chance-level decoding at 25% (horizontal red line). Error bars indicate ±1 SEM. Dotted horizontal grey lines indicate confidence intervals. Asterisks indicate when decoding accuracy was significantly above chance.

Fig. 5

Decoding accuracies during each session, shown as pairs of bar graphs, comparing the first four blocks with the second four blocks. Chance is at 25% (horizontal red line). The columns in dark/solid colours represent decoding accuracy over the first four blocks, averaged across all sessions; the lighter columns represent decoding accuracy over the second four blocks, averaged across all sessions. Decoding accuracy in bilateral LOC and bilateral parietal ROIs was significantly higher during the first half of each session, as compared to the second half of each session. Error bars indicate ±1 SEM. Dotted horizontal grey lines indicate confidence intervals. Asterisks indicate significant differences in decoding accuracy, comparing the first four with the second four blocks.

Fig. 6

Decoding accuracies for individual participants, comparing the first four blocks (Figure. A), with the second four blocks (Figure. B), averaged across all sessions. Chance is at 25% (horizontal red line). Dotted horizontal grey lines indicate confidence intervals.

Fig. 7

Graph showing average reaction times averaged across participants for n-back task performance, for each session. Sessions were divided further into the first 4 and second 4 blocks to show the effects of experimental time on task performance. Matched average reaction times for individual participants are shown for first 4 blocks and second 4 blocks of each session, using coloured connected lines for each participant.

Fig. 8

Proposed pipeline using a non-invasive BCI interface with rt- fMRI to prime and prepare specific brain regions with a BCI task, prior to surgery for placement of longer-term implantable BCI. 1) Realtime-fMRI decoding pathway (e.g. as used in this study) 2) A. Implantation of subdural electrodes (Image courtesy of Anna Miserocchi and Andrew McEvoy) B, C. 3D reconstruction showing final placement of temporal and inferior temporal subdural (ECoG) grids for recording of relevant cortical activity, as part of a long-term implanted BCI.