Evaluation of variation in preclinical electroencephalographic (EEG) spectral power across multiple laboratories and experiments: An EQIPD study

Abstract

The European Quality In Preclinical Data (EQIPD) consortium was born from the fact that publications report challenges with the robustness, rigor, and/or validity of research data, which may impact decisions about whether to proceed with further preclinical testing or to advance to clinical testing, as well as draw conclusions on the predictability of preclinical models. To address this, a consortium including multiple research laboratories from academia and industry participated in a series of electroencephalography (EEG) experiments in mice aimed to detect sources of variance and to gauge how protocol harmonisation and data analytics impact such variance. Ultimately, the goal of this first ever between-laboratory comparison of EEG recordings and analyses was to validate the principles that supposedly increase data quality, robustness, and comparability. Experiments consisted of a Localisation phase, which aimed to identify the factors that influence between-laboratory variability, a Harmonisation phase to evaluate whether harmonisation of standardized protocols and centralised processing and data analysis reduced variance, and a Ring-Testing phase to verify the ability of the harmonised protocol to generate consistent findings. Indeed, between-laboratory variability reduced from Localisation to Harmonisation and this reduction remained during the Ring-Testing phase. Results obtained in this multicentre preclinical qEEG study also confirmed the complex nature of EEG experiments starting from the surgery and data collection through data pre-processing to data analysis that ultimately influenced the results and contributed to variance in findings across laboratories. Overall, harmonisation of protocols and centralized data analysis were crucial in reducing laboratory-to-laboratory variability. To this end, it is recommended that standardized guidelines be updated and followed for collection and analysis of preclinical EEG data.

Fig 1. Localisation phase total power analysed locally by the partners.

Tukey boxplots and individual data points for the log₁₀ total power values per genotype group for each participating laboratory during phase 1. Lab 2 and Lab 4 found significant reduction in total power in Tg4510 mice compared to WT controls (see S4 Table).

Fig 2. Localisation phase total power analysed locally by the partners.

The figure shows estimated means of TG-WT contrasts (genotype effect) with each the lower confidence limit (CL) and higher CL on log₁₀ total power data.

Fig 3. Localisation phase relative theta power analysed locally by the partners.

Tukey boxplots and individual data points for the log₁₀ relative theta power values (obtained as a percentage of total power for individual subjects) per genotype group and every participating laboratory during phase 1 data collections. Only Lab 2 found a significant increase in relative theta power in Tg4510 mice compared to WT controls (see S5 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 4. Localisation phase relative theta power analysed locally by the partners.

The figure shows estimated means of TG-WT contrasts (genotype effect) with each the lower confidence limit (CL) and higher CL on log₁₀ relative theta power data.

Fig 5. Localisation phase histology.

Variability in electrode placement depth and underlying cortical damage that occurred in few animals across multiple laboratories. Increasing damage from top left (good), top right (fair), bottom left (poor), to bottom right (worst), where surgery-induced lesions occurred not only in cortex but also in underlying CA1 due to screw electrode being inserted too deeply. Note, the best EEG implants only contact dura and do not make any damage.

Fig 6. Harmonisation phase total power analysed locally by the partners.

Tukey boxplots and individual data points for the log₁₀ total power values per genotype group and every participating laboratory during phase 2 data collections. Lab 2 and Lab 3 found a significant decrease in total power in Tg4510 mice compared to WT controls (see S6 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 7. Harmonisation phase total power analysed locally by the partners.

The figure shows estimated means of the genotypic TG-WT contrasts (“treatment effect”) for each the lower confidence limit (CL) and higher CL on log₁₀ total power data.

Fig 8. Harmonisation phase relative theta power analysed centrally.

Tukey box-plots and individual data points for the log₁₀ relative theta power values (obtained as a percentage of total power for individual subjects) per genotype group and every participating laboratory during phase 2 data collections. No laboratory found a significant difference in relative power for Tg4510 mice compared to WT controls. The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 9. Harmonisation phase relative theta power analysed centrally.

The figure shows estimated means of TG-WT contrasts with for each the lower confidence limit (CL) and higher CL on log₁₀ relative theta power data.

Fig 10. Harmonisation phase relative theta power analysed centrally with Lab 5 removed.

Tukey box-plots and individual data points for the log₁₀ relative theta power values (obtained as a percentage of total power for individual subjects) per genotype group and every participating laboratory during phase 2 data collections. The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 11. Harmonisation phase relative theta power analysed centrally with Lab 5 removed.

The figure shows estimated means of TG-WT contrasts for each the lower confidence limit (CL) and higher CL on log₁₀ relative theta power data for harmonisation phase 2 without Lab 5.

Fig 12. Ring-Testing phase relative gamma power analysed locally by the partners.

Tukey boxplots and individual data points for the log₁₀ relative gamma power per genotype and every participating laboratory during phase 3 data collections. Four of the 6 laboratories (Lab 1, Lab 3, Lab 5, and Lab 6) found a significant increase in relative gamma power following 0.2 mg/kg MK-801 compared to vehicle (see S8 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 13. Ring-Testing phase relative gamma power analysed locally by the partners.

The figure shows estimated means of treatment contrasts for each the lower confidence limit (CL) and higher CL on locally analysed log10 relative gamma power data.

Fig 14. Ring-Testing phase gamma power as percent change from baseline using raw power analysed locally by the partners.

The figure shows Tukey box-plots and individual data points for the raw gamma power percent change values per compound and every participating laboratory during phase 3 data collections. Four of the 6 laboratories (Lab 1, Lab 4, Lab 5, and Lab 6) found a significant increase in gamma power percent change following 0.2 mg/kg MK-801 compared to vehicle (see S9 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 15. Ring-Testing phase gamma power as percent change from baseline using raw power analysed locally by the partners.

The figure shows estimated means of pharmacological contrasts with for each the lower confidence limit (CL) and higher CL on raw gamma power percent change data.

Fig 16. Ring-Testing phase relative gamma power analysed centrally.

Tukey box-plots and individual data points for the log₁₀ relative gamma power values per compound and every participating laboratory during phase 3 data collections. All displayed data were analysed centrally. Four of the 6 laboratories (Lab 1, Lab 2, Lab 5, and Lab 6) found a significant increase in relative gamma power following 0.2 mg/kg MK-801 compared to vehicle (see S10 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 17. Ring-Testing phase relative gamma power analysed centrally.

The figure shows estimated means of pharmacological contrasts for each the lower confidence limit (CL) and higher CL on log₁₀ relative gamma power data.

Fig 18. Ring-Testing phase gamma power as percent change from baseline using raw power analysed centrally.

Tukey box-plots and individual data points for the raw gamma power percent change values per compound and every participating laboratory during phase 3 data collections. One of the 6 laboratories (Lab 6; Lab 2, p = 0.051) found a significant increase in relative gamma power following 0.2 mg/kg MK-801 compared to vehicle (see S11 Table). The boxplot displays individual data points, as well as the median, the first (Q1) and third (Q3) quartiles and the whiskers are based on the interquartile range (IQR; Q3 –Q1) where they are not higher than Q3 + 1.5 * IQR and lower than Q1–1.5 * IQR.

Fig 19. Ring-Testing phase gamma power as percent change from baseline using raw power analysed centrally.

The figure shows estimated means of pharmacological contrasts with for each the lower confidence limit (CL) and higher CL on log₁₀ raw gamma power percent change data.