Enhancing peptide identification confidence by combining search methods

Abstract

Confident peptide identification is one of the most important components in mass-spectrometry-based proteomics. We propose a method to properly combine the results from different database search methods to enhance the accuracy of peptide identifications. The database search methods included in our analysis are SEQUEST (v27 rev12), ProbID (v1.0), InsPecT (v20060505), Mascot (v2.1), X! Tandem (v2007.07.01.2), OMSSA (v2.0) and RAId_DbS. Using two data sets, one collected in profile mode and one collected in centroid mode, we tested the search performance of all 21 combinations of two search methods as well as all 35 possible combinations of three search methods. The results obtained from our study suggest that properly combining search methods does improve retrieval accuracy. In addition to performance results, we also describe the theoretical framework which in principle allows one to combine many independent scoring methods including de novo sequencing and spectral library searches. The correlations among different methods are also investigated in terms of common true positives, common false positives, and a global analysis. We find that the average correlation strength, between any pairwise combination of the seven methods studied, is usually smaller than the associated standard error. This indicates only weak correlation may be present among different methods and validates our approach in combining the search results. The usefulness of our approach is further confirmed by showing that the average cumulative number of false positive peptides agrees reasonably well with the combined E-value. The data related to this study are freely available upon request.

Figure 1

The statistical calibrations for centroid data. (A) The profile E-value (using calibration formulas(4)) versus the average of the cumulative number of false positives when tested using the centroid data (A5−A8 subsets of ref (19)). (B)The statistical calibration results but after (manually) removing highly homologous peptides from the hit list and after the method-specific factor a_method (see text) has been applied. (C and D) We apply the calibrated formula (along with the method-specific factor found from the first calibration) to the subsets A1−A4 and A9−A12 of ref (19). We find that the calibration done using subset A5−A8 (when applied to other centroid data subsets) provides us with realistic statistics, supporting the universality of statistical calibration. It is worth noting that the lowest E-value in those calibration plot can only go to roughly one over the total number of spectra used for calibration. Since we used about 10 000 spectra, the lowest E-value that can be shown is of order 10⁻⁴. In real database searches, a really significant hit probably may have an E-value much smaller than 10⁻⁴ and many users may not wish to consider hits with E-values larger than 10⁻¹.

Figure 2

ROC curves for the seven database search methods tested when using the centroid data (A1−A4 of ISB data set). Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A, while the ROC curves of the second type are displayed in panel B. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 600, corresponds approximately to E-value 0.1. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 3

Final P-value from combining a reported P-value P₂ and a fixed P-value P₁. The fixed P-value P₁ is chosen to be either 1, 10⁻¹, 10⁻², 10⁻³ or 10⁻⁴. As one may see, although the relationship between P₂ and the final P-value is still reasonably linear in the log − log plot, the slope has deviated from 1.

Figure 4

ROC curves for the seven pairwise combinations giving rise to seven largest AUC, values shown in panel A, when using the centroid data (A1−A4 subsets of the ISB data). Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A. Panel B shows ROC curves of the second type. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 600, corresponds approximately to E-value of 0.1. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 5

ROC curves for the seven triplets giving rise to seven largest AUC, values shown in panel A, when using the centroid data (A1−A4 subsets of the ISB data). Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A. Panel B shows ROC curves of the second type. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 600, corresponds approximately to E-value of 0.1. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 6

Method correlations evaluated using the centroid data (A1−A4 subsets of the ISB data). Each search method is abbreviated by its first letter in the figure legend. The panels on the first row display the RC ratio, CTP(E ≤ E_c)/CFP(E ≤ E_c), described in eq 11 as a function of the cutoff E-value. The panels on the second row display the likelihood of mistaking a common false hit as a significant hit, see eq 12.

Figure 7

Examination of the combined E-value when using the centroid data (A1−A4 subsets of the ISB data). In every panel, the average cumulative number of false hits is plotted against the combined E-value. Within the E-value range investigated, the final combined E-value is mostly within a factor of 5 of the theoretical value, represented by y = x lines. As before, each method is represented by its first letter in the figure legend.

Figure 8

ROC curves for the seven database search methods tested when using the profile data. Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A, while the ROC curves of the second type are displayed in panel B. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 3000, corresponds approximately to E-value of 0.4. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 9

ROC curves for the seven pairwise combinations giving rise to seven largest AUC, values shown in panel A, when using the profile data. Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A for the profile data. For the same data, panel B shows ROC curves of the second type. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 3000, corresponds approximately to E-value of 0.4. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 10

ROC curves for the seven triplets giving rise to seven largest AUC, values shown in panel A, when using the profile data. Each search method is abbreviated by its first letter in the figure legend. ROC curves of the first type are displayed in panel A. Panel B shows ROC curves of the second type. Since the total number of spectra in this subset is about 7000, in panel B, the displayed highest number of false positives, 3000, corresponds approximately to E-value of 0.4. We did not show ROC curves of the second type to larger FP value because users probably will not be too interested in the large E-value regime.

Figure 11

Method correlations evaluated using profile data. Each search method is abbreviated by its first letter in the figure legend. The panels on the first row display the RC ratio, CTP(E ≤ E_c)/CFP(E ≤ E_c), described in eq 11 as a function of the cutoff E-value. The panels on the second row display the likelihood of mistaking a common false hit as a significant hit, see eq 12.

Figure 12

Examination of the combined E-value when using the profile data. In every panel, the average cumulative number of false hits is plotted against the combined E-value. Within the E-value range investigated, the final combined E-value is mostly within a factor of 5 of the theoretical value, represented by y = x lines. As before, each method is represented by its first letter in the figure legend.