The 2012/2013 ABRF Proteomic Research Group Study: Assessing Longitudinal Intralaboratory Variability in Routine Peptide Liquid Chromatography Tandem Mass Spectrometry Analyses

Abstract

Questions concerning longitudinal data quality and reproducibility of proteomic laboratories spurred the Protein Research Group of the Association of Biomolecular Resource Facilities (ABRF-PRG) to design a study to systematically assess the reproducibility of proteomic laboratories over an extended period of time. Developed as an open study, initially 64 participants were recruited from the broader mass spectrometry community to analyze provided aliquots of a six bovine protein tryptic digest mixture every month for a period of nine months. Data were uploaded to a central repository, and the operators answered an accompanying survey. Ultimately, 45 laboratories submitted a minimum of eight LC-MSMS raw data files collected in data-dependent acquisition (DDA) mode. No standard operating procedures were enforced; rather the participants were encouraged to analyze the samples according to usual practices in the laboratory. Unlike previous studies, this investigation was not designed to compare laboratories or instrument configuration, but rather to assess the temporal intralaboratory reproducibility. The outcome of the study was reassuring with 80% of the participating laboratories performing analyses at a medium to high level of reproducibility and quality over the 9-month period. For the groups that had one or more outlying experiments, the major contributing factor that correlated to the survey data was the performance of preventative maintenance prior to the LC-MSMS analyses. Thus, the Protein Research Group of the Association of Biomolecular Resource Facilities recommends that laboratories closely scrutinize the quality control data following such events. Additionally, improved quality control recording is imperative. This longitudinal study provides evidence that mass spectrometry-based proteomics is reproducible. When quality control measures are strictly adhered to, such reproducibility is comparable among many disparate groups. Data from the study are available via ProteomeXchange under the accession code PXD002114.

Fig. 1.

Timeline for the data generated across all 63 of the participating laboratories from March 2012 to January 2013 (participant 914061 is divided into four groups). A total of 526 MS data files were generated during the study. Note: color coding is included to readily visualize the different laboratories.

Fig. 2.

Robust PCA plot for the 458 data files from the 45 participating laboratories that submitted a minimum of eight raw data files. The first two principal components account for 36.5% of the variability, and ∼60% of the variability in the data is accounted for by the first five principal components. Data is color-coded according to mass spectrometer type.

Fig. 3.

Dissimilarity measures between pairs of experiments at each of the 45 participating laboratories that submitted a minimum of eight raw data files (participant 914061 is divided into four groups). Asterisks represent extreme pair-wise distances within the data, i.e. outside Q1–1.5 IQR or Q3 + 1.5 IQR.

Fig. 4.

χ²-quality control chart based on T²-statistics. The filled pink and blue circles represent analyses that are outlier and within control experiments, respectively. The family error rate used for each participant was 0.01. Filled pink dots encircled in black are isolated outlying experiments, which were not considered in the detection of change points. Asterisks indicate outliers that were identified by dissimilarity measures. Batch means identified from the change point analysis are indicated by the black horizontal segments.

Fig. 5.

Spectral counts identified in each file from the 45 laboratories that submitted a minimum of eight raw data files. Data is color-coded according to mass spectrometer type.

Fig. 6.

Coefficient of variation based on spectral counts and quality metrics. SPC, spectral counts.