Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier

Abstract

Fully automated machine learning methods based on structural magnetic resonance imaging (MRI) data can assist radiologists in the diagnosis of Alzheimer's disease (AD). These algorithms require large data sets to learn the separation of subjects with and without AD. Training and test data may come from heterogeneous hardware settings, which can potentially affect the performance of disease classification. A total of 518 MRI sessions from 226 healthy controls and 191 individuals with probable AD from the multicenter Alzheimer's Disease Neuroimaging Initiative (ADNI) were used to investigate whether grouping data by acquisition hardware (i.e. vendor, field strength, coil system) is beneficial for the performance of a support vector machine (SVM) classifier, compared to the case where data from different hardware is mixed. We compared the change of the SVM decision value resulting from (a) changes in hardware against the effect of disease and (b) changes resulting simply from rescanning the same subject on the same machine. Maximum accuracy of 87% was obtained with a training set of all 417 subjects. Classifiers trained with 95 subjects in each diagnostic group and acquired with heterogeneous scanner settings had an empirical detection accuracy of 84.2±2.4% when tested on an independent set of the same size. These results mirror the accuracy reported in recent studies. Encouragingly, classifiers trained on images acquired with homogenous and heterogeneous hardware settings had equivalent cross-validation performances. Two scans of the same subject acquired on the same machine had very similar decision values and were generally classified into the same group. Higher variation was introduced when two acquisitions of the same subject were performed on two scanners with different field strengths. The variation was unbiased and similar for both diagnostic groups. The findings of the study encourage the pooling of data from different sites to increase the number of training samples and thereby improving performance of disease classifiers. Although small, a change in hardware could lead to a change of the decision value and thus diagnostic grouping. The findings of this study provide estimators for diagnostic accuracy of an automated disease diagnosis method involving scans acquired with different sets of hardware. Furthermore, we show that the level of confidence in the performance estimation significantly depends on the size of the training sample, and hence should be taken into account in a clinical setting.

Fig. 1

Flow-chart representation of the comparison of an example computation to the mean accuracy distribution of random sets. Comparison of PURE set accuracy with expected accuracy of mixed sets using z-test.

Fig. 2

Box-plots of leave-one-out cross-validation (LOO-CV) accuracy as function of group size (x-axis) obtained by 500 random permutations.

Fig. 3

Changes of the SVM decision value (y-axis) between back-to-back scans (BTB), separately for diagnostic group (first panel) and field strength (second panel), and with changes between two field strengths (thus also two systems), separately for diagnostic group (third panel) for acquisitions of 96 subjects. Change in field strength (FS) does not introduce systematic bias (one-sample t-test p>0.05). A: Training set composed of 316 images. B: Training set composed of 80 images.

Fig. 4

Variance of decision values when comparing back-to-back scans or change of field strength compared to the effect of group. A: Training set size=316. B: Training set size=80. CN: healthy controls, AD: subject with probable AD, BTB: back-to-back (within subjects), FS: field strength (within subjects).