Behavioral Classification of Sequential Neural Activity Using Time Varying Recurrent Neural Networks

Abstract

Shifts in data distribution across time can strongly affect early classification of time-series data. When decoding behavior from neural activity, early detection of behavior may help in devising corrective neural stimulation before the onset of behavior. Recurrent neural networks are common models for sequence data. However, standard recurrent neural networks are not able to handle data with temporal distributional shifts to guarantee robust classification across time. To enable the network to utilize all temporal features of the neural input data, and to enhance the memory of recurrent neural networks, this paper proposes a novel approach: recurrent neural networks with time-varying weights, here termed Time-varying recurrent neural networks. These models are able to not only predict the class of the time-sequence correctly, but also lead to accurate classification earlier in the sequence than standard recurrent neural networks, while also stabilizing gradient dynamics. This paper focuses on early sequential classification of spatially distributed neural activity across time using Time-varying recurrent neural networks applied to a variety of neural data from mice and humans, as subjects perform motor tasks. Time-varying recurrent neural networks detect self-initiated lever-pull behavior up to 6 seconds before behavior onset-3 seconds earlier than standard recurrent neural networks. Finally, this paper explored the contribution of different brain regions on behavior classification using SHapley Additive exPlanation value, and found that the somatosensory and premotor regions play a large role in behavioral classification.

Fig. 1.

(A) In the WFCI dataset, mice were trained to pull a lever for water reward, while WFCI activity was recorded from multiple regions. (B) Neural activity of healthy and PD human subjects in a grip force task was recorded using fMRI.

Fig. 2.

(A) Standard RNNs and (B) Time-Varying RNNs (TV-RNNs) used for behavioral classification of neural activity from different brain regions.

Fig. 3.

Plot of an example (A) simulated ‘behavior’ trial, (B) WFCI ‘behavior’ trial, (C) fMRI ‘behavior’ trial. Short Time Fourier Transform (STFT) magnitude of (D) simulated behavior signal, (E) WFCI dataset and (F) fMRI dataset.

Fig. 4.

(A) Temporal classification accuracy curve of standard RNNs and TV-RNNs using simulated data. The stars on top represent the earliest decoding time for each model (see Methods), and the bars on the right side reflect the final classification accuracy of the sequence. Note that chance accuracy level is 0.5 for both datasets. Gradients of an example recurrent weight during training (first 1000 epochs) in (B) RNN-S2 and (C) TV-RNN show that TV-RNNs are able to reduce the probability of gradient vanishing and exploding (see Methods). All other recurrent weights have similar gradients plot (not shown). (D) Histogram of gradients during training shows that TV-RNN has less small gradients, i.e., gradient vanishing.

Fig. 5.

(A) Determining the window size $w$ of TV-RNN: area under curve and earliest decoding time (see Methods) while varying $w$ from 6 to 30; triangles represent standard RNN. (B) Temporal accuracy of standard RNNs with two training strategies and TV-RNNs, the stars depict the earliest decoding time with the height representing the sequential classification accuracy. (C) Histogram of the area under accuracy curve using standard RNNs and TV-RNNs for all sessions of mouse. (D) Histogram of the earliest decoding time using standard RNNs and TV-RNNs for all sessions of mouse.

Fig. 6.

(A) Temporal accuracy of behavioral classification between ‘force’ and ‘rest’ for PD patients; (B) Temporal accuracy of behavioral classification between ‘force’ and ‘rest’ for healthy control.

Fig. 7.

(A) Output trajectories of standard RNNs (average across trials), in the WFCI data. The shaded region provides the standard deviation. (B) Similarly, the output trajectories of TV-RNNs.

Fig. 8.

(A) Euclidean distance between $W_h^t$ of TV-RNN at different time. (B) Euclidean distance between $W_x^t$ of TV-RNN at different time. (C) Euclidean distance between $W_y^t$ of TV-RNN at different time.

Fig. 9.

(A)(B)(C) Importance matrix of three example sessions with TV-RNN. The color represents importance based on SHAP values, with darker color indicating higher importance and lighter color indicating lower importance.

Fig. 10.

(A) Importance matrix of PD subjects with TV-RNNs; (B) Importance matrix of healthy subjects with TV-RNN.