ADHD-200 Global Competition: diagnosing ADHD using personal characteristic data can outperform resting state fMRI measurements

Abstract

Neuroimaging-based diagnostics could potentially assist clinicians to make more accurate diagnoses resulting in faster, more effective treatment. We participated in the 2011 ADHD-200 Global Competition which involved analyzing a large dataset of 973 participants including Attention deficit hyperactivity disorder (ADHD) patients and healthy controls. Each participant's data included a resting state functional magnetic resonance imaging (fMRI) scan as well as personal characteristic and diagnostic data. The goal was to learn a machine learning classifier that used a participant's resting state fMRI scan to diagnose (classify) that individual into one of three categories: healthy control, ADHD combined (ADHD-C) type, or ADHD inattentive (ADHD-I) type. We used participants' personal characteristic data (site of data collection, age, gender, handedness, performance IQ, verbal IQ, and full scale IQ), without any fMRI data, as input to a logistic classifier to generate diagnostic predictions. Surprisingly, this approach achieved the highest diagnostic accuracy (62.52%) as well as the highest score (124 of 195) of any of the 21 teams participating in the competition. These results demonstrate the importance of accounting for differences in age, gender, and other personal characteristics in imaging diagnostics research. We discuss further implications of these results for fMRI-based diagnosis as well as fMRI-based clinical research. We also document our tests with a variety of imaging-based diagnostic methods, none of which performed as well as the logistic classifier using only personal characteristic data.

Keywords: ADHD; ICA; children; classifier; diagnosis; functional connectivity; machine learning; multivoxel pattern analysis.

Figure 1

Flow chart summarizing fMRI data preprocessing, normalization, dimensionality reduction, feature extraction, and testing in diagnosis tasks.

Figure 2

Top: Accuracies on the binary diagnosis task (controls vs. ADHD) using fMRI input data after dimensionality reduction/feature extraction with the T Avg, PCA₁, PCA₁₋₅, FFT, ALFF, FC_DMN, or FC₁₋₂₀ procedures (section 2.5) with five different classifiers (section 2.2). Each blue dot with error bars shows the mean accuracy and standard deviation for one Diagnostic Pipeline (feature data and classifier combination) on 10-fold cross validation with the 668 participant Training Dataset. The black horizontal line at 64.2% accuracy indicates chance baseline (guessing healthy control for all participants). The red horizontal line at 75.0% accuracy indicates the best mean accuracy achieved with binary diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Bottom: Accuracies on the three-way diagnosis task (controls vs. ADHD-C vs. ADHD-I) using same Diagnostic Pipelines as in top panel. The red horizontal line at 69.0% accuracy indicates the best mean accuracy achieved with three-way diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Other conventions as in top panel.

Figure 3

Top: Accuracies on the binary diagnosis task (controls vs. ADHD) using features derived from clusters of significant difference on group comparisons (section 2.6) with five different classifiers (section 2.2). Each blue dot with error bars shows the mean accuracy and standard deviation for one Diagnostic Pipeline (feature data and classifier combination) on 10-fold cross validation with the 668 participant Training Dataset. The black horizontal line at 64.2% accuracy indicates chance baseline (guessing healthy control for all participants). The red horizontal line at 75.0% accuracy indicates the best mean accuracy achieved with binary diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Bottom: Accuracies on the three-way diagnosis task (controls vs. ADHD-C vs. ADHD-I) using same Diagnostic Pipelines as in top panel. The red horizontal line at 69.0% accuracy indicates the best mean accuracy achieved with three-way diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Other conventions as in top panel.

Figure 4

Top: Accuracies on the binary diagnosis task (controls vs. ADHD) using features derived from the robust cluster identification procedure (section 2.6) with five different classifiers (section 2.2). Each blue dot with error bars shows the mean accuracy and standard deviation for one Diagnostic Pipeline (feature data and classifier combination) on 10-fold cross validation with the 668 participant Training Dataset. The black horizontal line at 64.2% accuracy indicates chance baseline (guessing healthy control for all participants). The red horizontal line at 75.0% accuracy indicates the best mean accuracy achieved with binary diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Bottom: Accuracies on the three-way diagnosis task (controls vs. ADHD-C vs. ADHD-I) using the same Diagnostic Pipelines as in top panel. The red horizontal line at 69.0% accuracy indicates the best mean accuracy achieved with three-way diagnosis on the Training Dataset using personal characteristic input data (section 3.1). Other conventions as in top panel.

Figure 5

Results from functional connectivity group analysis. Top-left: A midline sagittal slice from a statistical map of voxels with significant weighting for the default mode network (DMN) in 668 participants. Red regions were positively weighted on the DMN. Blue regions were negatively weighted. Front of brain is on the right side of the image. Colored bar indicates t-value scaling. The slice's coordinate in mm in MNI space is shown in the upper-left corner. All results p < 0.05 corrected for multiple comparisons. Middle-left: Axial slice of same DMN weighting map as shown in upper-left panel. Left side of brain is on left side of image. Other conventions as in top-left panel. Top-right: Sagittal slice from a contrast map comparing DMN weighting for 239 ADHD patients vs. 429 healthy controls. Red regions showed greater DMN weighting in patients, whereas blue regions showed greater weighting in controls. One cluster of significant difference, including parts of posterior cingulate cortex and thalamus, is outlined in green. Other conventions as in top-left panel. Middle-right: Axial slice from same contrast map as shown in upper-right panel. Conventions as in middle-left panel. Bottom: Histograms of patients' (red bars) and controls' (blue bars) average DMN weighting values across the voxels in the posterior cingulate/thalamus cluster which is outlined in green on the contrast maps. Group means are shown as thick vertical bars (red: patients, blue: controls). Though mean DMN weighting was significantly larger for controls than for patients in this region, there was substantial overlap among the weighting values for the participants' in the two groups.

Figure 6

Top-row: Axial slices of default mode network (DMN) localizer map and three patients vs. controls statistical comparison maps computed on different folds of 10-fold cross validation. Each fold omitted from the comparison a different subset of 66 or 67 participants out of the 668 participant Training Dataset. For the localizer map, red/blue regions exhibited weighting significantly above/below zero. For contrast maps, red and blue regions exhibited greater weighting for patients and for controls, respectively. Left side of axial slice represents left side of brain. Numbers in the upper-left corners of the images indicate slice coordinates in MNI space. Color bars show t-value scaling. Bottom-row: Equivalent to top-row expect that localizer and contrast maps were taken from a different functional connectivity (FC) weighting map. This map appears to reflect gray matter and white matter structure. It is noteworthy that most but not all clusters of significance were present in all three-folds depicted. In addition, cluster sizes and shapes differed to some extent on different folds.

Figure 7

Illustration of group differences vs. individual differences. Left and middle panels show simulated fMRI activation levels from an arbitrary brain region for two groups (blue and red) of 1000 participants each. Frequency histograms for participants' activation levels appear as pale blue and pale red bars. The actual Gaussian distributions from which participants were drawn are shown as dark blue and dark red curves. Black vertical rectangles (which may appear as thick black lines) show 95% confidence intervals of average activation across each group. Group comparisons between blue and red average activation levels are significant in both panels (left: p « machine precision, t = 78.7, df = 1998; middle: p = 2 × 10⁻¹², t = 7.1, df = 1998). The high statistical significance in both cases derives from the large number of participants, which allows us to estimate the means with high precision (Central Limit Theorem). There is little overlap between the blue and red groups in the left panel, and activation levels can predict individuals' groups with accuracy of 93%. In the middle panel, there is substantial overlap between the groups, and inferring participants' groups from their activation levels yields poor accuracy of 56% (compared to baseline chance accuracy of 50%). This illustrates that group differences with high statistical significance can, but do not necessarily, translate into good diagnostic criteria for distinguishing individuals from different groups. Right panel shows that combining activation levels from two regions, neither of which can separate the groups on its own because the marginal distributions overlap substantially, can allow good separation of participant groups if the joint distributions do not overlap much.

Figure A1

Figure shows sine waves (blue) with original sampling periods of 2.0 or 2.5 s as well as re-sampled curves (red) with re-sampling periods of 2.0 or 0.5 s. Red curves were linearly interpolated from blue curves. Re-sampling from 2.0 to 2.0 s (upper-left panel) leaves the curve unchanged. Re-sampling from 2.5 to 2.0 s (upper-right panel) causes re-sampling errors—the minimum and maximum values for the red curve are closer to zero than for the blue curve. Linear re-sampling from 2.0 or 2.5 s into 0.5 s (lower-left and lower-right panels) does not introduce such re-sampling errors, though of course it cannot recover sine wave values for time points falling in-between the originally-sampled time points.

Figure A2

Figure shows slices from the between-subject mean-normalized fMRI volume (see text). Red line indicates the manually-determined edge of the binary mask volume.