Brain-Inspired Spatio-Temporal Associative Memories for Neuroimaging Data Classification: EEG and fMRI

Abstract

Humans learn from a lot of information sources to make decisions. Once this information is learned in the brain, spatio-temporal associations are made, connecting all these sources (variables) in space and time represented as brain connectivity. In reality, to make a decision, we usually have only part of the information, either as a limited number of variables, limited time to make the decision, or both. The brain functions as a spatio-temporal associative memory. Inspired by the ability of the human brain, a brain-inspired spatio-temporal associative memory was proposed earlier that utilized the NeuCube brain-inspired spiking neural network framework. Here we applied the STAM framework to develop STAM for neuroimaging data, on the cases of EEG and fMRI, resulting in STAM-EEG and STAM-fMRI. This paper showed that once a NeuCube STAM classification model was trained on a complete spatio-temporal EEG or fMRI data, it could be recalled using only part of the time series, or/and only part of the used variables. We evaluated both temporal and spatial association and generalization accuracy accordingly. This was a pilot study that opens the field for the development of classification systems on other neuroimaging data, such as longitudinal MRI data, trained on complete data but recalled on partial data. Future research includes STAM that will work on data, collected across different settings, in different labs and clinics, that may vary in terms of the variables and time of data collection, along with other parameters. The proposed STAM will be further investigated for early diagnosis and prognosis of brain conditions and for diagnostic/prognostic marker discovery.

Keywords: EEG; NeuCube; STAM; fMRI; neuroimage classification; neuroimaging data; spatio-temporal associative memory; spiking neural networks.

Figure 1

Learning in SNN relates to changes in the connection weights between two spatially located spiking neurons over time so that both “time” and “space” are learned in the spatially distributed connections (http://en.m.wikipedia.org/wiki/neuron, accessed on 13 November 2023).

Figure 2

The NeuCube brain-inspired SNN architecture (from [6], (©Elsevier, reproduced with permission from Kasabov, N., NeuCube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data, Neural Networks, vol. 52, 2014)).

Figure 3

Original EEG signal (top), encoded into spike sequence (middle) and a reconstruction of the signal from the spike sequence, back to real values (bottom) (from [9], (©Springer-Nature 2019, reproduced with permission from Kasabov, N., Time-Space, Spiking Neural Networks and Brain-Inspired Artificial Intelligence, 2019)).

Figure 4

(a) Training the NeuCube STAM-EEG model on full data (60 EEG samples) and validating it on T1 = 80% of the time of the data (see Table 2). Different input neurons, representing corresponding EEG channels, are presented in different colors; (b) Post-training neuronal connectivity and cluster formations. (c) (Left): The size of the segments represents the spiking activity of the corresponding input neuron to an EEG channel; the largest the section, the higher the impact this channel has on the model; (Right): EEG electrode layout.

Figure 4

(a) Training the NeuCube STAM-EEG model on full data (60 EEG samples) and validating it on T1 = 80% of the time of the data (see Table 2). Different input neurons, representing corresponding EEG channels, are presented in different colors; (b) Post-training neuronal connectivity and cluster formations. (c) (Left): The size of the segments represents the spiking activity of the corresponding input neuron to an EEG channel; the largest the section, the higher the impact this channel has on the model; (Right): EEG electrode layout.

Figure 5

(a) Mapping of the 5062 fMRI voxels into a 3D SNN model of the NeuCube framework; (b) selecting top-20 voxels as input variables using SNR ranking (on the y-axis) of top voxels (on the x-axis) related to the affirmative versus negative sentences. The top features are selected according to their SNR values that were greater than a threshold = 0.4. (c) a full STAM-fMRI model implemented in NeuCube trained and tested on 100% of the data using all 20 features; (d) its training accuracy is 100%, but the validation association and generalization accuracies are further tested below.

Figure 6

(a) Three snapshots of learning of 8-s fMRI data in a STAM-fMRI model when a subject is reading a negative sentence (time in seconds); Positive connections are colored in blue and negative connections in red. (b) Internal structural pattern represented as spatio-temporal connectivity in the SNN model trained with 8-s fMRI data stream; (c) A functional pattern represented as a sequence of spiking activity of clusters of spiking neurons in a trained SNN model. The arrows show the order of activation of different spatially distributed neuronal areas after fMRI data is presented to an already trained SNNcube.

Figure 7

Parameters for spike encoding and validation of the STAM-fMRI model from Section 5.2. Left panel: For validation, only 70% (0.7) from the initial time points of the fMRI samples, equaled to 5.6-s data, are used, rather than using 8 s of the data for training the full model. Right panel: The model is tested/validated only on 50% of the temporal length (4 s) of the training data. The classification temporal association accuracy for both experiments is 100%. Using less than 50% of the time series results in an accuracy of less than 100%.

Figure 8

Classification spatial association accuracy is 100% when 18 input features are used. The panel at the write shows the correct classification of all input fMRI samples in class 1 (in green) and class 2 (in blue).

Figure 9

Distribution of the average connection weights around the input voxels located in the left and right hemispheres of the trained SNN models related to negative sentences (in (a)) and affirmative sentences (in (b)). The dominant voxels for the discrimination of the negative from the affirmative sentences are LDLPFC, LIPL, LT, and LSGA.