Real-time fMRI using brain-state classification

Abstract

We have implemented a real-time functional magnetic resonance imaging system based on multivariate classification. This approach is distinctly different from spatially localized real-time implementations, since it does not require prior assumptions about functional localization and individual performance strategies, and has the ability to provide feedback based on intuitive translations of brain state rather than localized fluctuations. Thus this approach provides the capability for a new class of experimental designs in which real-time feedback control of the stimulus is possible-rather than using a fixed paradigm, experiments can adaptively evolve as subjects receive brain-state feedback. In this report, we describe our implementation and characterize its performance capabilities. We observed approximately 80% classification accuracy using whole brain, block-design, motor data. Within both left and right motor task conditions, important differences exist between the initial transient period produced by task switching (changing between rapid left or right index finger button presses) and the subsequent stable period during sustained activity. Further analysis revealed that very high accuracy is achievable during stable task periods, and that the responsiveness of the classifier to changes in task condition can be much faster than signal time-to-peak rates. Finally, we demonstrate the versatility of this implementation with respect to behavioral task, suggesting that our results are applicable across a spectrum of cognitive domains. Beyond basic research, this technology can complement electroencephalography-based brain computer interface research, and has potential applications in the areas of biofeedback rehabilitation, lie detection, learning studies, virtual reality-based training, and enhanced conscious awareness.

Figure 1

Real‐time brain‐state imaging and voxel selection. Black arrows represent conventional fMRI: a stimulus is presented to a volunteer during image acquisition. (A) During training experiments (orange dashed line), the experimental condition (brain state) is used to label corresponding image times, and train a classifier on the scanner's image reconstruction hardware. (B) For training Runs 1 and 3 and for Run 2 (where the scanner was operated under testing mode, but no feedback was presented), volunteers were presented with the left button press condition or the right button press condition. For these runs, the arrow was always in the center of the visual field and oriented toward the target. (C) During testing with feedback (green dash‐dot line), the brain state is not known, but is estimated from each image during image reconstruction. The volunteer's brain state, then, is used as feedback to control the stimulus. (D) For the 4th feedback‐testing run, the stimulus target still alternated between left and right conditions. Shown is a specific example of a possible display update from one TR to the next. In this case, the goal is to move the arrow toward the left target. Given that the current display has the arrow directed to the left, a subsequent left code will advance the arrow toward the target. On the other hand, if a right code is sent from the MR scanner to the display computer, the arrow will change its direction. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]

Figure 2

Brain masking procedure to discard background, eyes, and tissue sources of high variance from fMRI data. A single image time volume is used to obtain an intensity threshold mask calculated from the first acquired image. In addition, the standard deviation of each voxel (using the entire run) is used to calculate the standard deviation mask. The final mask result is obtained by applying AND operator pixelwise to the two intermediate masks. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]

Figure 3

Visually guided motor experiment. (A) Classifier output for the feedback testing (Run 4) for four subjects (labeled as S1–S4), demonstrating consistently accurate results across subjects. (B) SVM map indicating relevant spatial locations for discriminating between left and right conditions for Subject 3, training Run 3. Using radiographic convention, the right hemisphere is shown on the left. Positive model values are displayed in red and negative in blue (This corresponds with the training convention that the left/right tasks are assigned −1/+1 class labels, respectively). (C) Learning curves for the four subjects generated with successively longer increments of training data. Mean (solid) +/− standard deviation (dashed) results were generated using the 12 possible train‐test permutations. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]

Figure 4

Classifier drift. (A) Classifier output without detrending for Subject 3, Run 4, training with Run 1 (thin line), Run 2 (medium line), and Run 3 (thick line), exemplifying significant offset and slight drift. (B) Classifier output for Subject 1, Run 3, using Run 1 as training data. The red line represents results after aligning images in Runs 1 and 3 to the first image in Run 1. The two black lines are uncorrected data, with and without detrended classifier output. The good correspondence between the aligned result and the detrended classifier output suggests alignment as an important factor contributing to this effect. (C) Classifier output for Subject 4, Run 4 (trained on Run 3), with and without detrending. (D) Subtraction of the two correlation maps from Run 4 using the time courses in (C). This result demonstrates discrepancies at the edges of the slices, suggesting that the drift in the raw classifier output arises from motion. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]

Figure 5

Hemodynamic effects. (A) Number of classification errors across all 16 left and right conditions for all 12 train‐test permutations for Subject 1 when training with all images and without the first one, two, or three transition images in each condition. (B) Classification accuracies for the 12 permutations of each case in (A), testing with all data in each run (black) and testing over the sustained activation of the last 20 s (10 images) in each experimental condition (gray). (C) Classification of transition (first 2 images for each 15 image condition) vs. “no transition” images (rather than left vs. right), obtained by aligning all four runs to the first image of the first run, using the first 3 runs as training data, and testing with Run 4. No correction was made for the unbalanced ratio (2:13) of training exemplars. (D) Time‐locked average of (C). (E) Average classifier output for two of the training conditions in (A)—using all images (closed circle) and excluding two transition images per task condition (closed triangle). (F) Classifier output with behavioral data showing an unaveraged example of the effect in (E). Output from the model that is not trained with the first two transition images in each condition is less responsive, lagging behind output from the full model. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]

Figure 6

Brain‐state classification across a variety of cognitive domains. With the exact same experimental setup (different instructions), subjects can learn to move the arrow with very high accuracy with an 8‐min training run, using (A) Mood (thinking happy vs. sad thoughts), (B) Language (bilingual subject progressing through a narrative, alternating between Mandarin and English), and (C) Imagined Motor (training with both the button press task and the imagined motor task, this subject transitioned to only the imagined motor task during the test run). [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.]