Test-retest reliability of freesurfer measurements within and between sites: Effects of visual approval process

Abstract

In the last decade, many studies have used automated processes to analyze magnetic resonance imaging (MRI) data such as cortical thickness, which is one indicator of neuronal health. Due to the convenience of image processing software (e.g., FreeSurfer), standard practice is to rely on automated results without performing visual inspection of intermediate processing. In this work, structural MRIs of 40 healthy controls who were scanned twice were used to determine the test-retest reliability of FreeSurfer-derived cortical measures in four groups of subjects-those 25 that passed visual inspection (approved), those 15 that failed visual inspection (disapproved), a combined group, and a subset of 10 subjects (Travel) whose test and retest scans occurred at different sites. Test-retest correlation (TRC), intraclass correlation coefficient (ICC), and percent difference (PD) were used to measure the reliability in the Destrieux and Desikan-Killiany (DK) atlases. In the approved subjects, reliability of cortical thickness/surface area/volume (DK atlas only) were: TRC (0.82/0.88/0.88), ICC (0.81/0.87/0.88), PD (0.86/1.19/1.39), which represent a significant improvement over these measures when disapproved subjects are included. Travel subjects' results show that cortical thickness reliability is more sensitive to site differences than the cortical surface area and volume. To determine the effect of visual inspection on sample size required for studies of MRI-derived cortical thickness, the number of subjects required to show group differences was calculated. Significant differences observed across imaging sites, between visually approved/disapproved subjects, and across regions with different sizes suggest that these measures should be used with caution.

Keywords: FreeSurfer; cerebral cortical surface area; cerebral cortical thickness; cerebral cortical volume; multisite MRI; test-retest reliability.

Figure 1

Flowchart summarizing the inspection process and pass/fail criteria, with example cases. Boxes represent image processing steps and examples of proper and inaccurate brain exaction outputs. See text for details. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 2

Histogram of cortical thickness for Destrieux atlas. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 3

Percent difference (PD) map for (from left to right) regional cortical thickness (CT), cortical volume (CV), and cortical surface area (CS), across the Desikan–Killiany atlas regions. Depicted PD values were averaged across the “approved” group subjects. Each column provides lateral (top) and medial (bottom) views. A green color scale was applied to convey the high reliability across these three outcome measures, evidenced by a relatively small and narrow range of PD values. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 4

(Top): Variation of percent difference (PD) of cortical thickness with respect to the mean cortical surface area (CS) across regions of DK atlas. Fit equation: ax^b, coefficients with confidence intervals: a = 11.46 (6.49, 16.43), b = −0.35 (−0.41, −0.29). Adjusted R ² of the fit = 0.63. (bottom): Variation of percent difference (PD) with respect to the mean cortical volume (CV) across regions of DK atlas. Fit equation: ax^b, coefficients with confidence intervals: a = 15.99 (5.47, 26.51), b = −0.35 (−0.43, −0.27). Adjusted R ² of the fit = 0.51. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 5

Median and the comparison intervals of TRC for cortical thickness (CT) (P = 0.003), cortical surface area (CS) (P < 0.001), and cortical volume (CV) (P = 0.041) in different sites. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 6

Median and the comparison intervals of ICC for cortical thickness (CT) (P < 0.001), cortical surface area (CS) (P = 0.008), and cortical volume (CV) (P = 0.405) in different sites. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 7

Median and the comparison intervals of PD for cortical thickness (CT) (P < 0.001), cortical surface area (CS) (P = 0.203), and cortical volume (CV) (P = 0.172) in different sites. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]