Behavioral Classification of Sequential Neural Activity Using Time Varying Recurrent Neural Networks

Abstract

Shifts in data distribution across time can strongly affect early classification of time-series data. When decoding behavior from neural activity, early detection of behavior may help in devising corrective neural stimulation before the onset of behavior. Recurrent Neural Networks (RNNs) are common models for sequence data. However, standard RNNs are not able to handle data with temporal distributional shifts to guarantee robust classification across time. To enable the network to utilize all temporal features of the neural input data, and to enhance the memory of an RNN, we propose a novel approach: RNNs with time-varying weights, here termed Time-Varying RNNs (TV-RNNs). These models are able to not only predict the class of the time-sequence correctly but also lead to accurate classification earlier in the sequence than standard RNNs. In this work, we focus on early sequential classification of brain-wide neural activity across time using TV-RNNs applied to a variety of neural data from mice and humans, as subjects perform motor tasks. Finally, we explore the contribution of different brain regions on behavior classification using SHapley Additive exPlanation (SHAP) value, and find that the somatosensory and premotor regions play a large role in behavioral classification.

Figure 1.

(A) In the WFCI dataset, mice were trained to pull a lever for water reward, while WFCI activity was recorded from multiple regions. (B) Neural activity of healthy and PD human subjects in a grip force task was recorded using fMRI.

Figure 2.

(A) standard RNN and (B) time varying RNN (TV-RNN) used for behavioral classification of neural activity from different brain regions.

Figure 3.

Plot of an example (A) simulated ‘behavior’ trial, (B) WFCI ‘behavior’ trial, (C) fMRI ‘behavior’ trial. Short Time Fourier Transform (STFT) magnitude of (D) simulated behavior signal, (E) WFCI dataset and (F) fMRI dataset.

Figure 4.

Temporal classification accuracy curve of standard RNNs and TV-RNNs using simulated data. The stars on top represent the earliest decoding time for each model (see Methods), and the bars on the right side reflect the final classification accuracy of the sequence. Note that chance accuracy level is 0.5 for both datasets.

Figure 5.

(A) Determining the window size w of TV-RNN: area under curve and earliest decoding time (see Methods) while varying w from 6 to 30; triangles represent standard RNN. (B) Temporal accuracy of standard RNNs with two training strategies and TV-RNNs, the stars depict the earliest decoding time with the height representing the sequential classification accuracy. (C) Histogram of the area under accuracy curve using standard RNNs and TV-RNNs for all sessions of mouse. (D) Histogram of the earliest decoding time using standard RNNs and TV-RNNs for all sessions of mouse.

Figure 6.

(A) Output trajectories of standard RNNs (average across trials), in the simulated data. The shaded region provides the standard deviation. (B) Similarly, the output trajectories of TV-RNNs. (C, D) Output trajectories for WFCI data: (C) standard RNNs; (D) TV-RNNs.

Figure 7.

(A) Euclidean distance between $W_h^t$ of TV-RNN at different time. (B) Euclidean distance between $W_x^t$ of TV-RNN at different time. (C) Euclidean distance between $W_y^t$ of TV-RNN at different time.

Figure 8.

(A) Temporal accuracy of behavioral classification between ‘force’ and ‘rest’ for PD patients; (B) Temporal accuracy of behavioral classification between ‘force’ and ‘rest’ for healthy control.

Figure 9.

(A) Importance matrix of simulated data with standard RNN; (B) Importance matrix of simulated data with TV-RNN.

Figure 10.

(A)(B)(C) Importance matrix of three example sessions with TV-RNN.

Figure 11.

(A) Importance matrix of PD subjects with TV-RNNs; (B) Importance matrix of healthy subjects with TV-RNN.