Empirical comparison of web-based antimicrobial peptide prediction tools

Abstract

Motivation: Antimicrobial peptides (AMPs) are innate immune molecules that exhibit activities against a range of microbes, including bacteria, fungi, viruses and protozoa. Recent increases in microbial resistance against current drugs has led to a concomitant increase in the need for novel antimicrobial agents. Over the last decade, a number of AMP prediction tools have been designed and made freely available online. These AMP prediction tools show potential to discriminate AMPs from non-AMPs, but the relative quality of the predictions produced by the various tools is difficult to quantify.

Results: We compiled two sets of AMP and non-AMP peptides, separated into three categories-antimicrobial, antibacterial and bacteriocins. Using these benchmark data sets, we carried out a systematic evaluation of ten publicly available AMP prediction methods. Among the six general AMP prediction tools-ADAM, CAMPR3(RF), CAMPR3(SVM), MLAMP, DBAASP and MLAMP-we find that CAMPR3(RF) provides a statistically significant improvement in performance, as measured by the area under the receiver operating characteristic (ROC) curve, relative to the other five methods. Surprisingly, for antibacterial prediction, the original AntiBP method significantly outperforms its successor, AntiBP2 based on one benchmark dataset. The two bacteriocin prediction tools, BAGEL3 and BACTIBASE, both provide very good performance and BAGEL3 outperforms its predecessor, BACTIBASE, on the larger of the two benchmarks.

Contact: gaberemu@ngha.med.sa or william-noble@uw.edu.

Supplementary information: Supplementary data are available at Bioinformatics online.

Fig. 1

Methodology employed in comparing and ranking various AMP tools

Fig. 2

Receiver operating characteristic curves for the 10 methods, separated by classification task. Panels on the left are for the DAMPD3 benchmark, and on the right are for APD3. For reference, each plot includes the line y = x, which corresponds to the performance of a random classifier. Each series is marked with a single point, indicating the location of the decision threshold selected by the method. In the key, the numeric values next to the name of each method are the corresponding ROC areas (Color version of this figure is available at Bioinformatics online.)

Fig. 3

Dependence of scores on sequence length (DAMPD dataset). In each panel, a point corresponds to an AMP or non-AMP peptide from the AMP (top row), antibacterial (middle row) or bacteriocin (bottom row) data set. The figures plot the score assigned to a peptide by a given prediction method as a function of the peptide length. Note that the BACTIBASE and BAGEL3 scores have been $-\text{log}\hspace{0.17em}$ transformed (Color version of this figure is available at Bioinformatics online.)

Fig. 4

Receiver operating characteristic for the AMP prediction task, with queries limited to 100 amino acids in length (DAMPD and APD3 datasets) (Color version of this figure is available at Bioinformatics online.)

Fig. 5

Scatter plot for prediction scores from AntiBP versus AntiBP2 (DAMPD dataset)