Challenges in diffusion MRI tractography - Lessons learned from international benchmark competitions

Abstract

Diffusion MRI (dMRI) fiber tractography has become a pillar of the neuroimaging community due to its ability to noninvasively map the structural connectivity of the brain. Despite widespread use in clinical and research domains, these methods suffer from several potential drawbacks or limitations. Thus, validating the accuracy and reproducibility of techniques is critical for sound scientific conclusions and effective clinical outcomes. Towards this end, a number of international benchmark competitions, or "challenges", has been organized by the diffusion MRI community in order to investigate the reliability of the tractography process by providing a platform to compare algorithms and results in a fair manner, and evaluate common and emerging algorithms in an effort to advance the state of the field. In this paper, we summarize the lessons from a decade of challenges in tractography, and give perspective on the past, present, and future "challenges" that the field of diffusion tractography faces.

Keywords: Accuracy; Algorithms; Challenges; Diffusion MRI; Tractography; Validation.

Figure 1.

Past challenges in fiber tractography. Detailed description of data, ground truth, and evaluation are described in the text. (A) FiberCup Phantom pathways with 16 ground truth bundles [21]. (B) Eight example CST reconstructions from the DTI Challenge [23]. (C) Synthetic fiber fields from the HARDI Reconstruction Challenge [26]. (D) Phantomas [27] dataset for Tractometer evaluation [28]. (E) Creation of simulated in vivo human dataset for the ISMRM Tractography Challenge [29]. (F) Example submissions from the TraCED Reproducibility Challenge for two white matter pathways. (G) 3D-VoTEM ground truths defined on the macaque, squirrel monkey, and phantom (from left to right). Reproduced and modified from Fillard et al. (2011), and Schilling et al. (2018) with permission from Elsevier; from Pujol et al. (2105) with permission from Wiley; from Daducci et al. (2014) with permission from IEEE, and from Maier-Hein et al. (2017) under a Create Commons license from Nature Publishing Group.

Figure 2.

Following local orientations, many algorithms are able to reconstruct valid connections. Images show the reconstructed fiber of all submissions (Methods #1–10) for each seed of the phantom (S1-S16). Variability across methods is apparent, and some pathways are more successful than others. Compare to Figure 1A for ground truth connections of each seed. Reproduced from Fillard et al. (2011) with permission from Elsevier.

Figure 3.

There are limitations to the use of tractography in clinical decision making - reconstruction of the CST results in a number of false positive and false negative connections. The figure shows eight tractography reconstructions of the pyramidal tract for patient 2 (top), patient 3 (center), and patient 4 (bottom). Each view presents the tracts (yellow: tumor side; orange: contralateral side) overlaid on axial and coronal T2-weighted image. Reproduced from Pujol et al. (2105) with permission from Wiley.

Figure 4.

Most reconstruction methods adequately resolve the fiber orientation distribution, even with limited data. A representative diffusion profile (e.g., ODF or FOD) as reconstructed by varying algorithms is shown for four different crossing configurations (90°, 60°, 45°, and 30°), with an SNR = 30. Reproduced from Daducci et al. (2014) with permission from IEEE.

Figure 5.

Challenges in tractography include bottle-necks and ambiguities cause by the ill-posed nature of tractography. (A) Example invalid bundles consistently identified in a majority of submissions, where tractography cannot differentiate the valid pathways due to the high amount of possible connections through a bottle-neck region. (B) For example, in the temporal lobe, six ground truth bundles converge in a parallel manner, resulting in more valid bundles per voxel than the number of unique peak directions, contributing to the tracking ambiguity. Reproduced and modified from Maier-Hein et al. (2017) under a Create Commons license from Nature Publishing Group.

Figure 6.

Tractography is reproducible within an algorithm, but highly variable across algorithms. (A) Overlay of all 46 Traced Challenge submissions from all sessions for several white matter tracts shows widespread spatial extent of various pathways. (B) Visualization of a single submission shows reasonable results for all pathways. White matter pathways include: A) Uncinate B) Fornix C) Cingulum D) Corticospinal tract E) Inferior Longitudinal Fasciculus F) Inferior Fronto-Occipital Fasciculus G) Superior Longitudinal Fasciculus H) Forceps major.

Figure 7.

Anatomical accuracy of tractography is limited. (A) Region-to-region connectivity validation is shown as ROC curves for the macaque (PCG seed and V4v seed) sub-challenge and squirrel monkey (M1 seed) sub-challenge. (B) Voxel-wise spatial overlap validation is shown with plots of overlap versus overreach for the squirrel monkey sub-challenge and phantom sub-challenge (scanner A and scanner B). One marker is shown for each submission, with marker colors indicating unique research groups. A common theme in this, and other challenges, is that a specificity/sensitivity (or overlap/overreach) tradeoff is inherent in all tractography algorithms and pipelines. Reproduced and modified from Schilling et al. (2018) with permission from Elsevier.