Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Jul 1;31(13):2174-81.
doi: 10.1093/bioinformatics/btv123. Epub 2015 Feb 25.

Automated benchmarking of peptide-MHC class I binding predictions

Thomas Trolle  1 , Imir G Metushi  1 , Jason A Greenbaum  1 , Yohan Kim  1 , John Sidney  1 , Ole Lund  1 , Alessandro Sette  1 , Bjoern Peters  1 , Morten Nielsen  2
Affiliations

Automated benchmarking of peptide-MHC class I binding predictions

Thomas Trolle et al. Bioinformatics. .

Abstract

Motivation: Numerous in silico methods predicting peptide binding to major histocompatibility complex (MHC) class I molecules have been developed over the last decades. However, the multitude of available prediction tools makes it non-trivial for the end-user to select which tool to use for a given task. To provide a solid basis on which to compare different prediction tools, we here describe a framework for the automated benchmarking of peptide-MHC class I binding prediction tools. The framework runs weekly benchmarks on data that are newly entered into the Immune Epitope Database (IEDB), giving the public access to frequent, up-to-date performance evaluations of all participating tools. To overcome potential selection bias in the data included in the IEDB, a strategy was implemented that suggests a set of peptides for which different prediction methods give divergent predictions as to their binding capability. Upon experimental binding validation, these peptides entered the benchmark study.

Results: The benchmark has run for 15 weeks and includes evaluation of 44 datasets covering 17 MHC alleles and more than 4000 peptide-MHC binding measurements. Inspection of the results allows the end-user to make educated selections between participating tools. Of the four participating servers, NetMHCpan performed the best, followed by ANN, SMM and finally ARB.

Availability and implementation: Up-to-date performance evaluations of each server can be found online at http://tools.iedb.org/auto_bench/mhci/weekly. All prediction tool developers are invited to participate in the benchmark. Sign-up instructions are available at http://tools.iedb.org/auto_bench/mhci/join.

Contact: mniel@cbs.dtu.dk or bpeters@liai.org

Supplementary information: Supplementary data are available at Bioinformatics online.

PubMed Disclaimer

Figures

Fig. 1.
Fig. 1.
Ranking scores for the initial IEDB benchmark. The scores for each server are calculated based on AUC performance, SRCC performance and both performance measures
Fig. 2.
Fig. 2.
The accumulated number of peptide-MHC measurements benchmarked by the automated benchmarking framework during its first 2 months. A total of 311 new measurements were identified and run during this time period
Fig. 3.
Fig. 3.
The number of unique alleles benchmarked by the automated benchmarking framework
Fig. 4.
Fig. 4.
The accumulated ranking score for each participating server, calculated after each weekly benchmark run during the first 2 months
Fig. 5.
Fig. 5.
Ranking scores calculated based on performance values from the dedicated dataset benchmark
Fig. 6.
Fig. 6.
A screenshot of the results page for the automated MHC-I benchmark. The individual dates may be clicked on to view detailed information on the evaluation datasets benchmarked that week
See this image and copyright information in PMC

References

    1. Androlewicz M.J., et al. . (1993) Evidence that transporters associated with antigen processing translocate a major histocompatibility complex class I-binding peptide into the endoplasmic reticulum in an ATP-dependent manner. Proc. Natl Acad. Sci. USA, 90, 9130–9134. - PMC - PubMed
    1. Bui H.-H., et al. . (2005) Automated generation and evaluation of specific MHC binding predictive tools: ARB matrix applications. Immunogenetics, 57, 304–314. - PubMed
    1. Cozzetto D., et al. . (2009) Evaluation of template-based models in CASP8 with standard measures. Proteins, 77(Suppl. 9), 18–28. - PMC - PubMed
    1. Eyrich V.A., et al. . (2001) EVA: continuous automatic evaluation of protein structure prediction servers. Bioinformatics, 17, 1242–1243. - PubMed
    1. Hattotuwagama C.K., et al. . (2004) Quantitative online prediction of peptide binding to the major histocompatibility complex. J. Mol. Graph. Model.,22, 195–207. - PubMed

Publication types