Real-time decision fusion for multimodal neural prosthetic devices

Abstract

Background: The field of neural prosthetics aims to develop prosthetic limbs with a brain-computer interface (BCI) through which neural activity is decoded into movements. A natural extension of current research is the incorporation of neural activity from multiple modalities to more accurately estimate the user's intent. The challenge remains how to appropriately combine this information in real-time for a neural prosthetic device.

Methodology/principal findings: Here we propose a framework based on decision fusion, i.e., fusing predictions from several single-modality decoders to produce a more accurate device state estimate. We examine two algorithms for continuous variable decision fusion: the Kalman filter and artificial neural networks (ANNs). Using simulated cortical neural spike signals, we implemented several successful individual neural decoding algorithms, and tested the capabilities of each fusion method in the context of decoding 2-dimensional endpoint trajectories of a neural prosthetic arm. Extensively testing these methods on random trajectories, we find that on average both the Kalman filter and ANNs successfully fuse the individual decoder estimates to produce more accurate predictions.

Conclusions: Our results reveal that a fusion-based approach has the potential to improve prediction accuracy over individual decoders of varying quality, and we hope that this work will encourage multimodal neural prosthetics experiments in the future.

Figure 1. Conceptual design of an artificial neural network.

(A) Each individual unit in the network accepts a weighted sum of input values, producing a single net activation value, n_j. (B) A three-layer network topology. This topology is feed-forward and fully-connected, that is, each unit links to all units in the layer directly after it.

Figure 2. Experimental design for fusion trials.

Flowchart describing fusion of Kalman filter (KF), PVA, and the optimal linear decodes using the Kalman filter and ANNs. Experimental trials contained three major phases: (i) individual decoder training, (ii) fusion decoder training, and (iii) final testing. In each experiment, individual decoders were first trained using the same simulated spike count data. Next, fusion decoders were trained on the individual decoders' outputs (predicted velocity components in x and y dimensions) for a separate fusion training dataset. An additional validation dataset was employed to prevent overtraining of ANNs. In final testing, trained individual decoders were used to predict the 2-d velocities, which were then compiled as input for fusion decoders. Endpoint velocity predictions from all decoders were then compared for accuracy. See Methods for details of the evaluation methodology.

Figure 3. Initial testing of fusion decoders.

(A) Decoded velocity trajectories for four trials. The true velocities are shown in red. The fused ANN and fused Kalman filter decodes are shown in brown and black, respectively. Individual decoders are plotted in varying shades of grey. (B) E_rms of 144 neural networks for four trial decodes. We examined a range of single and double hidden-layer networks to optimize the fusion results. Rows correspond to 1st-layer sizes, while columns are 2^nd-layer sizes. Note the first column in each matrix corresponds to all single hidden-layer networks. Interestingly, many single hidden-layer networks outperform more complex networks, indicating the dynamic accuracies of different neural network topologies. Table 2 displays the corresponding E_rms values for each decoder.

Figure 4. Fusion results of using potentially poor quality decoders.

These two sets correspond to trials 2 and 3 in Table 3. (A) Example trials showing individual and fusion decodes. True velocities are shown in red. The fused ANN and fused Kalman filter decodes are shown in brown and black, respectively. Individual decoders are plotted in varying shades of grey. (B) Corresponding pointwise root mean squared error of decodes over time. Note that time is unitless in these simulations. Though the decoders have variable accuracy over time, the fusion algorithms maintain acceptable decoding accuracy throughout the entire trials.

Figure 5. Results of decoders on 468 random trajectories (E_rms mean ± s.e.).

The improvement of fusion algorithms over the combined individual decoders was statistically significant (p<1e-29 in both cases, two-tailed Welch's T-test). While the fusion Kalman filter produced the significantly more accurate outputs than the individual decoders, the ANN limited to a single topology did not perform as well, illustrating an advantage of the Kalman filter as a fusion method.