Bridging big data in the ENIGMA consortium to combine non-equivalent cognitive measures

Abstract

Investigators in neuroscience have turned to Big Data to address replication and reliability issues by increasing sample size. These efforts unveil new questions about how to integrate data across distinct sources and instruments. The goal of this study was to link scores across common auditory verbal learning tasks (AVLTs). This international secondary analysis aggregated multisite raw data for AVLTs across 53 studies totaling 10,505 individuals. Using the ComBat-GAM algorithm, we isolated and removed the component of memory scores associated with site effects while preserving instrumental effects. After adjustment, a continuous item response theory model used multiple memory items of varying difficulty to estimate each individual's latent verbal learning ability on a single scale. Equivalent raw scores across AVLTs were then found by linking individuals through the ability scale. Harmonization reduced total cross-site score variance by 37% while preserving meaningful memory effects. Age had the largest impact on scores overall (- 11.4%), while race/ethnicity variable was not significant (p > 0.05). The resulting tools were validated on dually administered tests. The conversion tool is available online so researchers and clinicians can convert memory scores across instruments. This work demonstrates that global harmonization initiatives can address reproducibility challenges across the behavioral sciences.

Keywords: Harmonization; Item response theory; Mega analysis; Traumatic brain injury; Verbal learning.

Fig. 1

Comparing multisite data of total of trials scores before and after ComBat harmonization and adjustment for (a) CVLT, (b) RAVLT, and (c) HVLT. Results are sorted by median score per study. Variation in site medians were reduced after harmonization. Full details for all sites are available in Supplementary Fig. S1.

Fig. 2

Comparing proportions of memory items recalled before and after harmonization. Mean scores for each site (dots) are shown broken out by instrument (color) and item (Top: Trial 1 immediate free recall, Middle: Total sum of all Trials, Bottom: Long-delay free recall scores).

Fig. 3

Visualizing covariate effects on covariate unadjusted, harmonized scores. (a) Boxplots of scores stratified by group (TBI vs. control) and sex/gender indicated that males and those with history of TBI had significantly lower scores on average. Age-related declines (b) and the beneficial effects of education (c) on scores were consistent across all AVLTs.

Fig. 4

Visualizing and Validating Conversions. (a) Average scores as a function of individual ability are shown approximated as cubic polynomial fits for immediate, short, and long delay trials. Scores shown are not normed or T-scored. Horizontal lines of equivalent ability connect equivalent scores across tests, which facilitates the construction of crosswalks. (b) Scatter plot and fit to the sum of learning Trial scores for a subset of cases who were administered both the CVLT and RAVLT (n = 36). The confidence area of the dually assessed data is shown in blue and agrees with the derived crosswalk for CVLT- > RAVLT (n = 9362, black dotted line).

Fig. 5

The distribution of unadjusted ability scores are shown for each site, ranked by ability and color-coded by median age per site.