Decoding visual information from high-density diffuse optical tomography neuroimaging data

Abstract

Background: Neural decoding could be useful in many ways, from serving as a neuroscience research tool to providing a means of augmented communication for patients with neurological conditions. However, applications of decoding are currently constrained by the limitations of traditional neuroimaging modalities. Electrocorticography requires invasive neurosurgery, magnetic resonance imaging (MRI) is too cumbersome for uses like daily communication, and alternatives like functional near-infrared spectroscopy (fNIRS) offer poor image quality. High-density diffuse optical tomography (HD-DOT) is an emerging modality that uses denser optode arrays than fNIRS to combine logistical advantages of optical neuroimaging with enhanced image quality. Despite the resulting promise of HD-DOT for facilitating field applications of neuroimaging, decoding of brain activity as measured by HD-DOT has yet to be evaluated.

Objective: To assess the feasibility and performance of decoding with HD-DOT in visual cortex.

Methods and results: To establish the feasibility of decoding at the single-trial level with HD-DOT, a template matching strategy was used to decode visual stimulus position. A receiver operating characteristic (ROC) analysis was used to quantify the sensitivity, specificity, and reproducibility of binary visual decoding. Mean areas under the curve (AUCs) greater than 0.97 across 10 imaging sessions in a highly sampled participant were observed. ROC analyses of decoding across 5 participants established both reproducibility in multiple individuals and the feasibility of inter-individual decoding (mean AUCs > 0.7), although decoding performance varied between individuals. Phase-encoded checkerboard stimuli were used to assess more complex, non-binary decoding with HD-DOT. Across 3 highly sampled participants, the phase of a 60° wide checkerboard wedge rotating 10° per second through 360° was decoded with a within-participant error of 25.8±24.7°. Decoding between participants was also feasible based on permutation-based significance testing.

Conclusions: Visual stimulus information can be decoded accurately, reproducibly, and across a range of detail (for both binary and non-binary outcomes) at the single-trial level (without needing to block-average test data) using HD-DOT data. These results lay the foundation for future studies of more complex decoding with HD-DOT and applications in clinical populations.

Keywords: Decoding; Functional neuroimaging; High-density diffuse optical tomography; Retinotopy.

Figure 1:. Feasibility of visual decoding with HD-DOT data using a template matching strategy.

Results are illustrated for a typical decoding attempt in one participant. The participant viewed a checkerboard wedge flickering in either the left or the right visual hemifield, interspersed with rest periods, while being imaged by HD-DOT. Training data were block-averaged to define “templates” of expected brain activity for each stimulus condition (top panels). Spatial Pearson correlation coefficients were then computed between each template and the oxyhemoglobin signal map at each time point in the test data (bottom panel). The resulting correlation values were compared between templates and to a designated threshold value (here selected to be 0.25 on an ad hoc basis, but later optimized by ROC analysis in Figure 2) to determine the decoded stimulus condition at each time point, and this was compared to the actual stimulus state (middle panels).

Figure 2:. Sensitivity and specificity of binary retinotopic decoding with HD-DOT data.

(A) Taking data from 18 task runs in one participant, a single task run was used for construction of templates, and independent test trials were pooled across the remaining 17 runs. Plotting all trials by their correlations with each of the two templates reveals a clustering by trial type. Clusters can be separated by thresholds, which can be swept across the full range of possible values and optimized in a receiver operating characteristic (ROC) analysis. (B) ROC curves for the 3 possible binary classifications.

Figure 3:. Reproducibility of binary retinotopic decoding with HD-DOT across sessions in a highly sampled participant.

(A) The data from the participant who performed the checkerboard viewing task 18 times were used to evaluate the reproducibility of HD-DOT decoding across imaging sessions. The single task run used as training data for template construction was changed 17 times, and each time all the remaining runs were pooled as test data to conduct an ROC analysis, producing 18 ROC curves for each of the three binary classification problems. Different shades between red and black were used to plot different ROC curves to make the individual curves more discernible. (B) ROC analysis was also conducted to evaluate decoding with every possible pairing of a single training task run and a single test task run (i.e., without pooling test data across multiple runs). Areas under the curve (AUCs) for all these ROC analyses are plotted in matrices for each of the three possible pairwise classifications, illustrating the reproducibility of accurate binary retinotopic decoding with HD-DOT.

Figure 4:. Reproducibility of binary HD-DOT decoding across n=5 participants.

(A) Data was taken from 4 additional participants who performed the same checkerboard stimulus viewing task twice each. Decoding and ROC analysis were conducted for each participant using one task run for template construction and another task run as test data; resulting ROC curves are shown for all 5 participants. (B) Decoding and ROC analysis were also conducted using templates from one task run in one participant and test data from another run in any participant, across every possible pairing of training and test participants. AUC values along the diagonals of these matrices illustrate reproducibility and variability of within-participant decoding across multiple individuals, while off-diagonal values indicate the feasibility of inter-individual decoding.

Figure 5:. 18-way classification of visual stimuli with varying eccentricity within a single participant.

A participant was imaged using HD-DOT while watching a flickering checkerboard ring over 8 cycles of either periodic expansion or contraction through 18 concentric positions on a screen. A template matching strategy was again used to decode stimulus location at each time point in a test dataset, here using 18 template maps – 1 for each stimulus phase. (A) Each stimulus phase is assigned a color (as per the color wheel) and a position along the vertical axis in the plots of actual and decoded stimulus positions. Pearson correlation coefficients were calculated between each of the 18 templates and the oxyhemoglobin signal map at every time point in the test data, and are plotted on the two graphs at the bottom of this panel. The decoded stimulus corresponds to the template with the maximum correlation at each time point. (B) A permutation test was performed to evaluate the significance of this decoding using every one of the 12 possible pairings of the 4 task runs collected in this participant for training and testing. The mean decoding error obtained using true template sets was compared with a null distribution generated using 10 random permutations of the training data for every true decoding attempt (p<0.0083). (C) Mean decoding error across the 3 test runs for a single training run is plotted here as a function of true stimulus position, showing little variation in decoding performance with eccentricity.

Figure 6:. 36-way classification of rotating visual stimulus position within a single participant.

A participant was imaged using HD-DOT while watching a flickering checkerboard wedge rotating 10 times through 36 positions over 36 seconds per revolution. A template matching strategy was used to decode stimulus location, here generating a set of 36 templates (one for each phase of the stimulus) from one training task run and decoding stimulus position at every time point in an independent test task run. (A) Each stimulus phase is assigned a color (as per the color wheel) and a position along the vertical axis in the plots of actual and decoded stimulus positions. Pearson correlation coefficients were calculated between each of the 36 templates and the oxyhemoglobin signal map at every time point in the test data, and are plotted on the two graphs at the bottom of this panel. The decoded stimulus corresponds to the template with the maximum correlation at each time point. (B) A permutation test was used to evaluate the statistical significance of the decoding performance. Mean decoding error was calculated for every one of the 30 possible pairings of training and test data set across the 6 task runs performed by this participant. The resulting error distribution was compared with a null distribution generated using 10 random permutations of the training data for each true template set (p<0.0033). (C) Mean decoding error across the 5 test runs for a single training run is plotted as a function of true stimulus position, showing better decoding performance as the wedge rotates through the lower half of visual space.

Figure 7:. Reproducibility of complex, non-binary visual decoding with HD-DOT across n=3 participants.

Three participants viewed the same rotating checkerboard stimulus paradigm six times across two imaging sessions each. Decoding performance was then evaluated using every possible pairing of template and test task run across participants. (A) The mean absolute error for each decoding attempt illustrates both the reproducibility and variability of within-subject and inter-individual decoding across the imaging sessions and participants. (B) Permutation testing comparing decoding performance across all sessions and participants to a null distribution.