MindSeer: a portable and extensible tool for visualization of structural and functional neuroimaging data

Abstract

Background: Three-dimensional (3-D) visualization of multimodality neuroimaging data provides a powerful technique for viewing the relationship between structure and function. A number of applications are available that include some aspect of 3-D visualization, including both free and commercial products. These applications range from highly specific programs for a single modality, to general purpose toolkits that include many image processing functions in addition to visualization. However, few if any of these combine both stand-alone and remote multi-modality visualization in an open source, portable and extensible tool that is easy to install and use, yet can be included as a component of a larger information system.

Results: We have developed a new open source multimodality 3-D visualization application, called MindSeer, that has these features: integrated and interactive 3-D volume and surface visualization, Java and Java3D for true cross-platform portability, one-click installation and startup, integrated data management to help organize large studies, extensibility through plugins, transparent remote visualization, and the ability to be integrated into larger information management systems. We describe the design and implementation of the system, as well as several case studies that demonstrate its utility. These case studies are available as tutorials or demos on the associated website: http://sig.biostr.washington.edu/projects/MindSeer.

Conclusion: MindSeer provides a powerful visualization tool for multimodality neuroimaging data. Its architecture and unique features also allow it to be extended into other visualization domains within biomedicine.

Figure 1

System Architecture. An overview of the general architecture of MindSeer, showing the main classes. The system follows the Model-View-Controller design pattern, and is divided into 3 main parts (Display, Communications and Back-end). Yellow classes are controllers, brown are models, and blue are views. The white box in the Back-end encloses plugins: abstract classes that are subclassed for specific data types and visualization methods.

Figure 2

MindSeer Library. A screen capture showing the tree form of the MindSeer Library, where the Library is specified by an XML file defining the input/output file specifications for a given visualization. This particular library is organized by coordinate space (Magnet), subject (P163), type (Functional, Structural), file type (Image, Surface, Volume) and modality (fMRI, EEG). All of these metadata are internally stored in free form XML tags.

Figure 3

User Interface. This screen capture shows the default layout of MindSeer with the Library, Scene and Controls. It also shows how the plugins work together to generate a scene.

Figure 4

Visualization. Left. Functional and structural volume data shown in the Slice Viewport, which displays three orthogonal slices. fMRI (warm colors), and 2 related source localized EEG volumes (green and magenta) are simultaneously shown superimposed on the structural MRI. Right. The same data shown in the Surface viewport. In this case the fMRI is shown as red iso-surfaces and the EEG data are shown with a cutaway view.

Figure 5

Plotting Coordinates. Left. A surgical photograph with Cortical Stimulation Mapping (CSM) sites. Right. A cortical surface of the patient's own brain onto which we have plotted the CSM points. Veins are in blue, and provide landmarks that enable the user to more easily match the photograph with the surface than would be possible with sulci and gyri alone. Sites that have been found to be significant for language are colored green.

Figure 6

Query Results. Results of a query by our DXBrain application, displayed on a canonical brain. The colored spheres represent the locations of temporal lobe CSM sites in which male patients made semantic (as opposed to syntactic) language errors in response to cortical stimulation. The locations, sizes and colors of the spheres were passed to MindSeer via the information system.

Figure 7

fMRI Surface Coloring Plugin. The plugin colors a cortical surface based on activation found in the corresponding fMRI volume dataset.