Verification of predicted robustness and accuracy of multivariate analysis

Abstract

The assessment of accuracy and robustness of multivariate analysis of FDG-PET brain images as presented in [Markiewicz, P.J., Matthews, J.C., Declerck, J., Herholz, K., 2009. Robustness of multivariate image analysis assessed by resampling techniques and applied to FDG-PET scans of patients with Alzheimer's disease. Neuroimage 46, 472-485.] using a homogeneous sample (from one centre) of small size is here verified using a heterogeneous sample (from multiple centres) of much larger size. Originally the analysis, which included principal component analysis (PCA) and Fisher discriminant analysis (FDA), was established using a sample of 42 subjects (19 Normal Controls (NCs) and 23 Alzheimer's disease (AD) patients) and here the analysis is verified using an independent sample of 166 subjects (86 NCs and 80 ADs) obtained from the ADNI database. It is shown that bootstrap resampling combined with the metric of the largest principal angle between PCA subspaces as well as the deliberate clinical misdiagnosis simulation can predict robustness of the multivariate analysis when used with new datasets. Cross-validation (CV) and the .632 bootstrap overestimated the predictive accuracy encouraging less robust solutions. Also, it is shown that the type of PET scanner and image reconstruction method has an impact on such analysis and affects the accuracy of the verification sample.

Fig. 1

Top: Accuracy, sensitivity and specificity of the PCA/FDA discrimination analysis trained on the original sample of 42 subjects (grey) and verified on the ADNI sample (black). Middle: Accuracy of the ADNI sample with matched scanner type and reconstruction method to that of the original sample. Bottom: Model selection using CV, the .632 bootstrap, the largest principal angle between PCA subspaces (median angle shown) and deliberate clinical misdiagnosis simulation (shown are the median with the range of the dispersion of the accuracy distributions). The performance of the different metrics derived from the original 42-subject sample are compared with the accuracy of the model in the ADNI sample for each choice of the number of PCs.