Human-centred physical neuromorphics with visual brain-computer interfaces

Abstract

Steady-state visual evoked potentials (SSVEPs) are widely used for brain-computer interfaces (BCIs) as they provide a stable and efficient means to connect the computer to the brain with a simple flickering light. Previous studies focused on low-density frequency division multiplexing techniques, i.e. typically employing one or two light-modulation frequencies during a single flickering light stimulation. Here we show that it is possible to encode information in SSVEPs excited by high-density frequency division multiplexing, involving hundreds of frequencies. We then demonstrate the ability to transmit entire images from the computer to the brain/EEG read-out in relatively short times. High-density frequency multiplexing also allows to implement a photonic neural network utilizing SSVEPs, that is applied to simple classification tasks and exhibits promising scalability properties by connecting multiple brains in series. Our findings open up new possibilities for the field of neural interfaces, holding potential for various applications, including assistive technologies and cognitive enhancements, to further improve human-machine interactions.

Fig. 1. BCI setup.

Input data (shown is an example image of a handwritten digit “0” and a set of control parameters) are encoded in frequency division multiplexing. The frequency-encoded signal modulates the intensity of an LED light projected onto a white screen, which is observed by a participant. A 3-pole EEG device detects the steady-state visual evoked potential, with an active electrode placed at Oz (medial occipital electrode site) to capture the electric signal from the primary visual cortex, a reference electrode positioned above the left ear (M1 position), and a ground electrode located above the right ear (M2 position). The resulting normalized power spectrum density (NPSD) is utilized for image transfer or computational tasks.

Fig. 2. BCI image transmission.

Experimental results are shown for a handwritten digit “7” image. The first row shows the SSVEP normalized power spectrum density (NPSD), produced by frequency division multiplexing following Eq. (1), with f₀ = 12 Hz, measurement time 196 s, and bandwidth a 1 Hz, b 2 Hz, c 4 Hz, d 8 Hz, e 12 Hz, f 16 Hz. g is for 12 Hz bandwidth and a shorter measurement time of 16.3 s, while h is for 12 Hz bandwidth with a blindfold (showing only an alpha wave peak at 10 Hz), and i is a zoom of a from 12 to 13 Hz. The second row, j–q shows the reconstructed, gray-scale images corresponding to the data in the image directly above in the first. Each figure also shows the structural similarity index measure (SSIM) relative to the ground truth image, shown in (r).

Fig. 3. BCI physical neural network image classification.

Experimental results from a single classification experiment of handwritten digits “0” and “1''. a An example of input data, a grayscale 8 × 8 pixel digit “0''. b Measured EEG signal NPSD with three highlighted frequency intervals: the input image frequency-encoded as 64 equidistant frequencies in the [15.0, 15.5] Hz range; the control parameters (determined by a genetic algorithm) frequency-encoded as 64 frequencies in the [20.0, 20.5] Hz range; and the 128 intermodulation frequencies in the [35, 36] Hz range. c The decoded intermodulation signal in more detail; the blue curve is a magnification of the measured signal in (b), and the red curve is the synthetic (numerically simulated) data. d The readout probability distribution over the two classes “0” and “1” showing a correct classification (highest probability) for “0''.

Fig. 4. Single and multi-layer physical neural network classification.

a Schematic architecture of the two-layer PNN. b Classification probabilities for the single layer PNN applied to the Iris dataset with three classes. Correct classifications are indicated with gray bars. c Classification probabilities for the two-layer PNN applied to the same Iris dataset. All three classifications are now correct, and classification probabilities are significantly improved, from  ~50% or less, now up to close to  ~80%.

Fig. 5. Effect of attention on physical neural network classification and on the intermodulation (IM) frequency power.

a PNN classification probability (i.e., the power fraction in each frequency segment) and b intermodulation (IM) frequency power for the two layers (brain) PNN with six participants, each acting only as the second layer (the first layer is fixed, participant 1). Participants are asked to 'focus' (blue bars) attention on the light flicker or 'disrupt' (red bars) attention by mentally performing mathematical operations (number additions, subtractions, divisions) for the duration of the light flicker (200 s). In all cases, participants fixate on the illuminated area of the screen. Each participant was measured twice, several minutes apart, inverting the order of the 'focus' and 'disrupt' condition, so as to exclude a possible confounding effect of the temporal order in which the conditions were performed. We found that PNN classification accuracy (t(5) = 6.29, p = 0.00006) and the intermodulation frequency power (t(5) = 4.18, p = 0.002) were statistically significantly reduced during the 'disrupt' compared to the 'focus' condition. These results indicate that, indeed, human attention can directly modify the effectiveness of the multilayer brain connection and PNN computing efficiency.