Tractography passes the test: Results from the diffusion-simulated connectivity (disco) challenge

Abstract

Estimating structural connectivity from diffusion-weighted magnetic resonance imaging is a challenging task, partly due to the presence of false-positive connections and the misestimation of connection weights. Building on previous efforts, the MICCAI-CDMRI Diffusion-Simulated Connectivity (DiSCo) challenge was carried out to evaluate state-of-the-art connectivity methods using novel large-scale numerical phantoms. The diffusion signal for the phantoms was obtained from Monte Carlo simulations. The results of the challenge suggest that methods selected by the 14 teams participating in the challenge can provide high correlations between estimated and ground-truth connectivity weights, in complex numerical environments. Additionally, the methods used by the participating teams were able to accurately identify the binary connectivity of the numerical dataset. However, specific false positive and false negative connections were consistently estimated across all methods. Although the challenge dataset doesn't capture the complexity of a real brain, it provided unique data with known macrostructure and microstructure ground-truth properties to facilitate the development of connectivity estimation methods.

Keywords: Challenge; Connectivity; Diffusion MRI; Microstructure; Monte carlo simulation; Numerical substrates; Tractography.

Fig. 1.

Ground-truth test dataset composed of 11,032 numerical tubular fibres. (A) 3D rendering showing the synthetic white matter mask (gray) and the 16 ROIs (colors). (B) Trajectories of the fibres of the 26 bundles, each shown using a different color. (C-D) 3D mesh of the outer layer of numerical fibres.

Fig. 2.

Challenge submission results of the 14 participating teams (111 submissions). (A) Fraction of valid connectivity weight in pairs of regions connected in the ground-truth connectivity matrix. (B) Pearson correlation coefficient between the participant’s submitted matrices and the ground-truth connectivity matrix of the validation dataset. (C) The area under the ROC curve (AUC) computed from the submitted matrices and the ground-truth binary connectivity matrix. (D) The accuracy (fraction of correctly identified pairs of ROIs, out of 120) of the binarised submitted matrices, thresholded at 5% of their maximal value. Numbers indicate the submission indices of each team.

Fig. 3.

The test dataset’s ground-truth connectivity matrix (top left) and each team’s best-performing classification matrices. All matrices are symmetric, and the upper triangular matrices are normalized to sum to one. The 26 non-zero connections of the test dataset have weights ranging from 0.008 to 0.092.

Fig. 4.

Receiver Operating Characteristic (ROC) curves of the submitted matrix with the highest correlation for each team. The black dashed line shows the performance of a connectivity matrix with randomly generated weights. The corresponding area under the curve (AUC) is reported in the bottom right panel.

Fig. 5.

The test dataset’s ground-truth binary connectivity matrix (top left) and each team’s matrices. All matrices were thresholded at 5% of their maximal value. The light/dark green and light/dark red colours show the true positives/negatives and false positives/negatives, respectively. All matrices are symmetric.

Fig. 6.

Percentage of classification error for each pair of ROIs for the submitted matrices (111) and using the threshold at 5% of their maximal value. The left subfigure reports the false positive connections. Regions 5–11 and 4–6 show the worst performance, with 73% (81) and 71% (79) matrices erroneously identifying them connected. The right subfigure reports the false negative connections. Regions 6–9, 4–16, and 3–14 show the worst classification, with 100% (111), 97% (108) and 95% (105) of methods erroneously identifying them as not connected. Both matrices are symmetric.

Fig. 7.

False positive bundles connecting ROIs 5–11 (A, blue) and 4–6 (B, green). These 2 pairs of regions have been incorrectly identified as connected by 73% and 71% of the submitted matrices, using a threshold at 5% of their maximal value, respectively. Glyphs show the local orientations of the ground-truth tubular fibres intersecting voxels, coloured with their orientation (left-right: red, anterior-posterior: green, superior-inferior: blue). Both pairs of regions are spatially located next to each other.

Fig. 8.

False negative bundles connecting ROIs 6–9 (green), 4–16 (red), and 3–14 (blue), were erroneously reported non-connected by 100%, 97% and 95% of methods, respectively. A) show a 3D rendering of the ground-truth fibre trajectories of the three bundles. B) and C) show a 2D cross-sectional image of the local orientations of the ground-truth tubular fibres, with fibre segment intersecting the 2D plane. All three bundles show a long and straight configuration going through the centre of the phantom and mixing with the other bundles. Those three bundles are the bundle with the lowest, second lowest and 5th lowest connectivity in the ground-truth weighted connectivity matrix.