Efficient emotion recognition using hyperdimensional computing with combinatorial channel encoding and cellular automata

Abstract

In this paper, a hardware-optimized approach to emotion recognition based on the efficient brain-inspired hyperdimensional computing (HDC) paradigm is proposed. Emotion recognition provides valuable information for human-computer interactions; however, the large number of input channels (> 200) and modalities (> 3 ) involved in emotion recognition are significantly expensive from a memory perspective. To address this, methods for memory reduction and optimization are proposed, including a novel approach that takes advantage of the combinatorial nature of the encoding process, and an elementary cellular automaton. HDC with early sensor fusion is implemented alongside the proposed techniques achieving two-class multi-modal classification accuracies of > 76% for valence and > 73% for arousal on the multi-modal AMIGOS and DEAP data sets, almost always better than state of the art. The required vector storage is seamlessly reduced by 98% and the frequency of vector requests by at least 1/5. The results demonstrate the potential of efficient hyperdimensional computing for low-power, multi-channeled emotion recognition tasks.

Keywords: Brain-inspired; Emotion recognition; Hardware efficient; Hyperdimensional computing; Memory optimization; Multi-modal sensor fusion; Wearable.

Fig. 1

Sensor fusion datapath from electrodes to a fused hypervector for the three-modality emotion recognition system used in AMIGOS with GSR, ECG, and EEG sensor inputs. The sensor inputs are pre-processed into a set of features which are then mapped into the HD space to create a set of spatial encoder (SE) inputs. These vectors are encoded within each modality, and then finally fused together to create one vector representing information from all of the channels. The process is detailed in Sect. 2

Fig. 2

HDC early fusion detailed architecture for m modalities with the four main blocks: map into HDS, spatial encoder, temporal encoder for n-gram of size n+1, and associative memory. Sensor fusion occurs early in the datapath, directly after the spatial encoder

Fig. 3

HDC a late fusion and b early fusion architectures for a three-modality emotion recognition system. The late fusion architecture fuses after the temporal encoder, resulting in 3 parallel temporal encoders—one per modality. In comparison, the early fusion architecture fuses before the temporal encoder, resulting in only 1 temporal encoder

Fig. 4

iM and FP vectors used to map into HDS to generate unique SE vectors per channel for a ‘unoptimized’ with distinct iM and FP vectors for all channels, b ‘iM vectors constant per modality’ with the same iM vectors between different modalities, and c ‘FP constant per feature channel’ with the same FP vectors between channels of the same modality

Fig. 5

‘Combinatorial pairs’ feature channel vector set generation demonstrated for 7 stored vectors. iM loops through vector bank after exhausting available sequential pairs for FP. Hybrid method follows by burst re-generating the vector bank with rule 90 so that new combinatorial pairs can be formed for more feature channels. Generation of 18 feature channel vector sets using a bank of only 7 vectors is shown

Fig. 6

Arousal and valence accuracies and required vector storage for the various memory optimization as compared to unoptimized for a AMIGOS and b DEAP data sets. Optimizations include ‘unoptimized’ with distinct iM and FP vectors for all channels, ‘iM vectors constant per modality’ with the same iM vectors between different modalities, ‘FP constant per feature channel’ with the same FP vectors between channels of the same modality, and ‘Rule 90 generation’ with generated FP and iM vectors on top of the previous optimizations

Fig. 7

Number of unique feature channel vector sets {iM, PFP, NFP} that can be generated, and hence the number of channels that can be encoded, using the combinatorial pair technique as the number of stored vectors increases

Fig. 8

Vector request rate (number of vectors requested per input channel) as the number of vectors stored increases while using rule 90 on a small set of continuously re-populating vectors as compared to only rule 90 for a AMIGOS and b DEAP

Fig. 9

Average valence and arousal accuracy for the various memory optimizations proposed as the hypervector length decreases for a AMIGOS and b DEAP. The data labels are shown for the most optimized version: rule 90 generation