Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls

Abstract

Neuroimaging-based single subject prediction of brain disorders has gained increasing attention in recent years. Using a variety of neuroimaging modalities such as structural, functional and diffusion MRI, along with machine learning techniques, hundreds of studies have been carried out for accurate classification of patients with heterogeneous mental and neurodegenerative disorders such as schizophrenia and Alzheimer's disease. More than 500 studies have been published during the past quarter century on single subject prediction focused on a multiple brain disorders. In the first part of this study, we provide a survey of more than 200 reports in this field with a focus on schizophrenia, mild cognitive impairment (MCI), Alzheimer's disease (AD), depressive disorders, autism spectrum disease (ASD) and attention-deficit hyperactivity disorder (ADHD). Detailed information about those studies such as sample size, type and number of extracted features and reported accuracy are summarized and discussed. To our knowledge, this is by far the most comprehensive review of neuroimaging-based single subject prediction of brain disorders. In the second part, we present our opinion on major pitfalls of those studies from a machine learning point of view. Common biases are discussed and suggestions are provided. Moreover, emerging trends such as decentralized data sharing, multimodal brain imaging, differential diagnosis, disease subtype classification and deep learning are also discussed. Based on this survey, there is extensive evidence showing the great potential of neuroimaging data for single subject prediction of various disorders. However, the main bottleneck of this exciting field is still the limited sample size, which could be potentially addressed by modern data sharing models such as the ones discussed in this paper. Emerging big data technologies and advanced data-intensive machine learning methodologies such as deep learning have coincided with an increasing need for accurate, robust and generalizable single subject prediction of brain disorders during an exciting time. In this report, we survey the past and offer some opinions regarding the road ahead.

Keywords: Brain disorders; Classification; Machine learning; Neuroimaging; Prediction.

Figure 1

Comparison of group difference analysis and classification in three different scenarios using toy data. Group difference is analyzed by two-sample t-tests and classification is performed by simple thresholding (red dotted lines). Each group/class has 100 samples. A: Significant group difference (p-value<0.001) but poor classification (60.0%). B: Insignificant group difference (p-value=0.865) but high classification accuracy (94.5%). C: Significant group difference (p-value<2e-16) and high classification accuracy (93.0%). Significant group difference doesn't necessarily cause high classification and vice versa.

Figure 2

The literature review procedure, the inclusion criteria and the number of surveyed studies for each modality.

Figure 3

Visual summary of Table 2-6. A: Total number of papers for two-year intervals for each modality. The inset legend shows the color code for each disorder. This legend also applies to figures in part B and C. B: Number of publications per modality for each disorder C: Scatter plot of overall reported accuracy versus the total sample size. D: Histogram of number of samples used in the surveyed studies. Vertical dashed lines show mean (red) and median (blue) sample size among all studies, which are 186 and 88 respectively. E: Disorder specific histograms of reported accuracies of all surveyed papers. Red dashed line indicates the mean accuracy. Black curves represent the estimated distribution of overall accuracy based on kernel density estimation.

Figure 4

Confusion matrix and common performance measures for binary classification. Measures such as sensitivity, specificity, precision, accuracy and F1 score are easily computable based on the four elements of the confusion matrix.

Figure 5

An example to show the effect of SVM hyperparameter optimization on classification accuracy for linear, polynomial and RBF kernels. Top row: un-optimized, Bottom row: optimized. Since the underlying pattern is non-linear, SVM with linear kernel fails to perform well in both scenarios. Performance of SVM with both polynomial and RBF kernels significantly improve when the parameters are optimized.