Effect of data harmonization of multicentric dataset in ASD/TD classification

Abstract

Machine Learning (ML) is nowadays an essential tool in the analysis of Magnetic Resonance Imaging (MRI) data, in particular in the identification of brain correlates in neurological and neurodevelopmental disorders. ML requires datasets of appropriate size for training, which in neuroimaging are typically obtained collecting data from multiple acquisition centers. However, analyzing large multicentric datasets can introduce bias due to differences between acquisition centers. ComBat harmonization is commonly used to address batch effects, but it can lead to data leakage when the entire dataset is used to estimate model parameters. In this study, structural and functional MRI data from the Autism Brain Imaging Data Exchange (ABIDE) collection were used to classify subjects with Autism Spectrum Disorders (ASD) compared to Typical Developing controls (TD). We compared the classical approach (external harmonization) in which harmonization is performed before train/test split, with an harmonization calculated only on the train set (internal harmonization), and with the dataset with no harmonization. The results showed that harmonization using the whole dataset achieved higher discrimination performance, while non-harmonized data and harmonization using only the train set showed similar results, for both structural and connectivity features. We also showed that the higher performances of the external harmonization are not due to larger size of the sample for the estimation of the model and hence these improved performance with the entire dataset may be ascribed to data leakage. In order to prevent this leakage, it is recommended to define the harmonization model solely using the train set.

Keywords: ABIDE; Autism spectrum disorder; Harmonization; Machine learning; Multi-site data.

Fig. 1

Effect of external (center) and internal (right) harmonization approaches on a structural features, left hemisphere cortical thickness, compared with the non-harmonized (left) scenario. The boxplots display the distributions of the features, grouped by site, which are sorted by increasing median age

Fig. 2

Effect of external (center) and internal (right) harmonization approaches on a connectivity features, 1385, compared with the not harmonized (left) scenario. The boxplots display the distributions of the features, grouped by site, which are sorted by increasing median age

Fig. 3

The ASD/TD classification results are reported, for different feature sets and harmonization strategies

Fig. 4

Structural (A–B) and connectivity (C–D) features of minSC dataset. Comparison between the no-harmonization and the internal-harmonization criteria (A–C) and between the external-harmonization and the internal-harmonization criteria (B–D). We show the 30% of the most important features for the internal-harmonization method in order to see what are the feature importance values of the same features for the no-harmonization method