The challenge of mapping the human connectome based on diffusion tractography

Abstract

Tractography based on non-invasive diffusion imaging is central to the study of human brain connectivity. To date, the approach has not been systematically validated in ground truth studies. Based on a simulated human brain data set with ground truth tracts, we organized an open international tractography challenge, which resulted in 96 distinct submissions from 20 research groups. Here, we report the encouraging finding that most state-of-the-art algorithms produce tractograms containing 90% of the ground truth bundles (to at least some extent). However, the same tractograms contain many more invalid than valid bundles, and half of these invalid bundles occur systematically across research groups. Taken together, our results demonstrate and confirm fundamental ambiguities inherent in tract reconstruction based on orientation information alone, which need to be considered when interpreting tractography and connectivity results. Our approach provides a novel framework for estimating reliability of tractography and encourages innovation to address its current limitations.

Fig. 1

Overview of synthetic data set. The top row summarizes the phantom generation process. The simulated images are generated from 25 major bundles, which are shown in the bottom part of the figure. These were manually segmented from a whole-brain tractogram of a HCP subject and include the CC, cingulum (Cg), fornix (Fx), anterior commissure (CA), optic radiation (OR), posterior commissure (CP), inferior cerebellar peduncle (ICP), middle cerebellar peduncle (MCP), superior cerebellar peduncle (SCP), parieto-occipital pontine tract (POPT), cortico-spinal tract (CST), frontopontine tracts (FPT), ILF, UF, and SLF. The connectivity plot in the middle shows the phantom design. The segment positions correspond to the involved endpoint region (from top to bottom: frontal lobe, temporal lobe, parietal lobe, occipital lobe, subcortical region, cerebellum, brain stem). The radial segment length and the connection number in the plot are chosen according to the volume of the respective bundle endpoint region. Abbreviations: right (R) and left (L) hemisphere, head (H) and tail (T) of each respective bundle

Fig. 2

Summary of teams and tractography pipeline setups. a Location of the teams’ affiliated labs. b Configuration of the different pipelines used for processing (A: motion correction, B: rotation of b-vectors, C: distortion correction, D: spike correction, E: denoising, F: upsampling, G: diffusion model beyond diffusion tensor imaging (DTI), H: tractography beyond deterministic, I: anatomical priors, J: streamline filtering, K: advanced streamline filtering, L: streamline clustering)

Fig. 3

Tractography identifies most of the ground truth bundles, but not their full extent. a Overview of scores reached for different bundles in ground truth. Average overlap (OL) and average overreach (OR) scores for the submissions (red: very hard, green: hard, blue: medium, for abbreviations see Fig. 1). b Representative bundles for DTI deterministic (DET) tracking come from submission 6/team 20, high angular resolution diffusion imaging (HARDI) deterministic tracking from submission 0/team 9, and HARDI probabilistic (PROBA) tracking from submission 2/team 12 (see Supplementary Note 5 for a discussion of these submissions). The first column shows ground truth VBs for reference. The reported OL and OR scores correspond to the highest OL score reached within the respective class of algorithms

Fig. 4

Between-group differences in tractography reconstructions of VBs and IBs. Overview of the scores reached by the different teams as a percentage of streamlines connecting valid regions, b number of detected VBs and IBs (data points are jittered to improve legibility), and c volume overlap (OL) and overreach (OR) scores averaged over bundles. Black arrows mark submissions used in the following figures (see Supplementary Note 5 for discussion)

Fig. 5

Overview of VBs and IBs and examples of invalid streamline clusters. a Each entry in the connectivity matrix indicates the number of submissions that have identified the respective bundle. The two rows and columns of each bundle represent the head-endpoint and tail-endpoint regions. The connectivity matrix indicates a high number of existing tracts that were identified by most submissions (red). It also indicates systematic artefactual reconstructions across teams (blue). b Examples of IBs that have been consistently identified by more than 80% of the submissions, but do not exist in the ground truth data set. The AF, for example, was generated from ILF and SLF crossing streamlines, whereas the IFOF was generated from by crossing ILF and UF streamlines. The MdLF, FAT, SFOF, and VOF were other examples of highly represented IBs

Fig. 6

Tractography on ground truth directions with no noise still affected by IB problem. We applied deterministic tractography directly to the ground truth vector field with multiple resolutions (2, 1.75, 1.5, 1.25, 1.0, 0.75, and 0.5 mm). Two independent implementations of deterministic tractography methods were used to obtain the results (GT₁ and GT₂, cf. Supplementary Note 2). a Percentage of streamlines connecting valid regions. b Number of detected VBs and IBs (data points are jittered to improve legibility). c Volume overlap and overreach scores averaged over bundles

Fig. 7

Ambiguous correspondences between diffusion directions and fiber geometry. The three illustrations at voxel, local, and global level are used as an example to illustrate the possible ambiguities contained in the diffusion imaging information that may lead to alternative tract reconstructions. (A) The intra-voxel crossing of fibers in the hypothetical ground truth leads to ambiguous imaging information at voxel level. (B) Similarly, the imaging representation of local fiber crossings can be explained by several other configurations. (C) At a global level, white matter regions that are shared by multiple bundles (so-called “bottlenecks”, dotted rectangles) can lead to many spurious tractographic reconstructions. With only two bundles in the hypothetical ground truth (red and yellow bundle), four potential false-positive bundles emerge. Please note that the hypothetical ground truth used in the global-level example is anatomically incorrect as most of the callosal fibers are homotopically distributed (i.e., connect similar regions on both hemispheres)

Fig. 8

Bottlenecks and the fundamental ill-posed nature of tractography. a Visualization of six ground truth bundles converging into a nearly parallel funnel in the bottleneck region of the left temporal lobe (indicated by square region). The bundles per voxel (box “# Valid bundles”) clearly outnumber the peak directions in the diffusion signal (box “# Signal peaks”). b Visualization of streamlines from a HCP in vivo tractogram passing through the same region. c Exemplary IBs that have been identified by more than 50% of the submissions, showing that tractography cannot differentiate between the high amount of plausible combinatorial possibilities connecting different endpoint regions (see Supplementary Movie 1). d Automatically QuickBundle-clustered streamlines from the in vivo tractogram going through the temporal ROI. The clustered bundles are illustrated in different shades of green. These clusters represent a mixture of true-positive and false-positive bundles going through that bottleneck area of the HCP data set (see Supplementary Movie 2)

Fig. 9

Illustration of artifacts included in the synthetic data set. Exemplary illustration of the spike (a), N/2 ghost (b), and distortion artifacts (c) contained in the final diffusion-weighted data set. Supplementary Movie 3 gives an impression of the complete synthetic data set provided