A Virtual Reality Visualization Tool for Neuron Tracing

Abstract

Tracing neurons in large-scale microscopy data is crucial to establishing a wiring diagram of the brain, which is needed to understand how neural circuits in the brain process information and generate behavior. Automatic techniques often fail for large and complex datasets, and connectomics researchers may spend weeks or months manually tracing neurons using 2D image stacks. We present a design study of a new virtual reality (VR) system, developed in collaboration with trained neuroanatomists, to trace neurons in microscope scans of the visual cortex of primates. We hypothesize that using consumer-grade VR technology to interact with neurons directly in 3D will help neuroscientists better resolve complex cases and enable them to trace neurons faster and with less physical and mental strain. We discuss both the design process and technical challenges in developing an interactive system to navigate and manipulate terabyte-sized image volumes in VR. Using a number of different datasets, we demonstrate that, compared to widely used commercial software, consumer-grade VR presents a promising alternative for scientists.

Fig. 1

A screenshot of our VR neuron tracing tool using the isosurface rendering mode. The dark gray floor represents the extent of the tracked space. Users can orient themselves in the dataset via the minimap (right), which shows the world extent in blue, the current focus region in orange, and the previously traced neuronal structures. The focus region is displayed in the center of the space. The 3D interaction and visualization provides an intuitive environment for exploring the data and a natural interface for neuron tracing, resulting in faster, high-quality traces with less fatigue reported by users compared to existing 2D tools.

Fig. 2

The technology probe and prototype were used to explore different interaction and rendering possibilities for scientific visualization and neuron tracing in VR.

Fig. 3

The wand model shown in VR can be changed from the physical model. On the tracing wand (a), we removed the top loop seen in (b) to avoid occlusion while tracing. The button sticking out underneath (a) is the trigger, and the large circular button is the trackpad. The icosphere brush in (a) is colored to match the selected line color.

Fig. 4

From left to right: the neuron tracing process begins by finding a neuron. A starting point is placed by moving the brush inside the neuron and pressing the trigger. While holding the trigger, the user follows the neuron with the brush, tracing it. To end the line, the trigger is released.

Fig. 5

A branch can be created by placing the brush close to an existing line, where a candidate branch point will be shown (a), or an existing node, and tracing from it. The branch can also be started as a new line and re-connected to the parent tree (b), in which case the candidate branch point created by the connection is shown.

Fig. 6

The anatomy of a single frame. WaitGetPoses blocks until ≈ 2ms before VSync and returns the latest head tracking data. This allows the renderer to start submitting work before VSync to fully utilize the GPU. We first submit draw calls for the geometry and volume, and then page in asynchronously uploaded volume data into the sparse texture.

Fig. 7

Examples of different mistakes and their effect on the DIADEM score. Trees (b) and (c) are compared against the reference (a) with scores 0.875 and 0.5, respectively. The error in (c) misses a large subtree, impacting later analysis more significantly than that in (b).

Fig. 8

Differences between scores of expert traces in VR vs. NeuroLucida. For each neuron traced, we compute the difference in score achieved compared to the reference between the two tools. We find that overall experts performed within the acceptable error range (±0.1, dark blue) and sometimes better in VR (light blue) when compared to their work in NeuroLucida.

Fig. 9

Distribution of scores (higher is better) for experts. In (a) median score: 0.7, mean score: 0.57 ± 0.38. In (b) median score: 0.6, mean score 0.49 ± 0.39. A score of ≥ 0.8 is a tracing acceptably similar to the reference.

Fig. 10

Distribution of scores (higher is better) for novices, excluding user 1. In (a) median score: 0.5, mean score: 0.42 ±0.37. In (b) median score: 0.49, mean score 0.5 ± 0.37. A score of ≥ 0.8 is a tracing acceptably similar to the reference.

Fig. 11

A stitching issue clearly visible in NeuroLucida (a–b), but difficult to perceive with volume rendering or isosurfacing (c). What appears as two neurons (c) is in fact a single neuron, slightly misaligned due to stitching issues at the border of two acquisitions. When scrolling through the image slices in NeuroLucida, the stitching issue can be seen by flipping between the slices (a–b) and those above and below. In NeuroLucida, all experts traced the neuron correctly, whereas in VR only one expert traced it correctly. We note that this issue is not specific to VR, but to the volume visualization method chosen.

Fig. 12

A neuron branching along the Z plane is not visible on the image plane used to trace the main structure (left). The branch can be seen only after scrolling down the stack (right). Only two experts traced this branch correctly in NeuroLucida, but in VR all users traced it correctly.

Fig. 13

From left to right: a neuron travels vertically through consecutive slices, appearing as a dot (middle) in these images. In NeuroLucida, only two experts traced this correctly, but in VR all users except one expert traced it correctly.