Horizon, closing the gap between cinematic visualization and medical imaging

Abstract

Medical imaging has become a fascinating field with detailed visualizations of the body's internal environments. Although the field has grown fast and is sensitive to new technologies, it does not use the latest rendering techniques available in other domains, such as day-to-day movie production or game development. In this work, we bring forward Horizon, a new engine that provides cinematic rendering capabilities in real-time for quality controlling medical data. In addition, Horizon is provided as free, open-source software to be used as a foundation stone for building the next generation of medical imaging applications. In this introductory paper, we focus on the extensive development of advanced shaders, which can be used to highlight untapped features of the data and allow fast interaction with machine learning algorithms. In addition, Horizon provides physically-based rendering capabilities, the epitome of advanced visualization, adapted for the needs of medical imaging analysis practices.

Keywords: Medical imaging; medical visualization; physically-based rendering; shader programming.

Figure 1.

Diagram of the distribution of microfacets on smooth and rough surfaces. The sphere on the left simulates two physical properties. On top is a smooth surface, and on the bottom is a rough surface. The first zoomed-in section (center) illustrates how microfacets might align on such surfaces. The rightmost section shows two different occlusion effects present on rough surfaces.

Figure 2.

Demonstration of Sheen and Clear Coat effects. On the left, we see the base PBR colors. On the right, we see how Sheen (top) and Clear Coat (bottom) will change the base color. To clarify the advantage, we can see the effects isolated, assuming the base colors are zero (black) or together with the actual base colors (colored). Notice how some effects, such as Clear Coat Glossiness, have a smaller magnitude and others, such as Sheen Tint, have a more profound effect. When combined with the base color, such effects can provide more information about the underlying geometry of the surface.

Figure 3.

Gallery of physically-based glass-like shading with absorption parameter. Base contains a glass-like surface with reflected and refracted colors extracted using Image Based Lighting (IBL). Then, three different levels of absorption (1, 3, and 5) are used to combine the colors from the IBL with a statistical map (top) and with interpolated colors from the anatomical structure of the brain (bottom).

Figure 4.

Example of a tree obtained from QuickBundlesX (QBX). On the left, the entire tree of centroids for a whole brain tractogram. For clarity, only the distance levels 30mm, 25mm, and 20mm are shown. On the right, a close-up of the subtrees A (orange) and B (purple) represent the left and right Corticospinal Tract, respectively.

Figure 5.

Horizon allows interaction with labeled tractograms directly. For example, a user starts by loading a full brain tractogram (A) which gets automatically clustered into bundles (centroids shown in B). Then, the user can expand any selected centroid (shown in C) or decide to hide the other clusters/centroids (shown in D). That process (cluster/selection/hide) can be repeated multiple times to remove subclusters (see red arrow) to obtain refined results, as shown in E.

Figure 6.

Visualization customization using physically-based shading. E.g., In A, atlas regions are projected on a cortical surface. B and C display a connectivity network alongside the cortical surface. In B, the surface opacity is reduced until the graph becomes visible. While in C, sheen highlights the surface’s silhouette without compromising the opacity. This effect preserves depth information, as seen in the marked regions (red circles).

Figure 7.

Comparison of lighting and coloring techniques on three views of the midsection of the Corpus Callosum. In each view, bundle with a plain gray color (A). Bundle with a plain gray color and anisotropic lighting (B). Bundle with colors interpolated from each fiber direction (C). And bundle with interpolated colors and anisotropic lighting (D).

Figure 8.

Comparison of fMRI statistical maps. A cortical surface with Horizon’s glass shader and IBL (left). Then, two different levels of absorption (3 and 5) (center) are used to combine the colors from the IBL with a statistical map. Last, the same statistical maps and interpolated colors on non-glass surfaces (right).

Figure 9.

Performance comparison between QuickBundles (QB) (Garyfallidis et al., 2012) and QuickBundlesX (QBX). Lower execution times mean faster clustering time. QBX is about 20X faster than QB, which was already a fast method for clustering streamlines.

Figure 10.

This figure demonstrates two advanced interfaces. On the left is an orbital menu that can be attached to any 3D object. On the right is a virtual reality application that uses OpenVR integration to render stereo image pairs on a mobile phone for use in a VR headset.