Systematically benchmarking peptide-MHC binding predictors: From synthetic to naturally processed epitopes

Abstract

A number of machine learning-based predictors have been developed for identifying immunogenic T-cell epitopes based on major histocompatibility complex (MHC) class I and II binding affinities. Rationally selecting the most appropriate tool has been complicated by the evolving training data and machine learning methods. Despite the recent advances made in generating high-quality MHC-eluted, naturally processed ligandome, the reliability of new predictors on these epitopes has yet to be evaluated. This study reports the latest benchmarking on an extensive set of MHC-binding predictors by using newly available, untested data of both synthetic and naturally processed epitopes. 32 human leukocyte antigen (HLA) class I and 24 HLA class II alleles are included in the blind test set. Artificial neural network (ANN)-based approaches demonstrated better performance than regression-based machine learning and structural modeling. Among the 18 predictors benchmarked, ANN-based mhcflurry and nn_align perform the best for MHC class I 9-mer and class II 15-mer predictions, respectively, on binding/non-binding classification (Area Under Curves = 0.911). NetMHCpan4 also demonstrated comparable predictive power. Our customization of mhcflurry to a pan-HLA predictor has achieved similar accuracy to NetMHCpan. The overall accuracy of these methods are comparable between 9-mer and 10-mer testing data. However, the top methods deliver low correlations between the predicted versus the experimental affinities for strong MHC binders. When used on naturally processed MHC-ligands, tools that have been trained on elution data (NetMHCpan4 and MixMHCpred) shows better accuracy than pure binding affinity predictor. The variability of false prediction rate is considerable among HLA types and datasets. Finally, structure-based predictor of Rosetta FlexPepDock is less optimal compared to the machine learning approaches. With our benchmarking of MHC-binding and MHC-elution predictors using a comprehensive metrics, a unbiased view for establishing best practice of T-cell epitope predictions is presented, facilitating future development of methods in immunogenomics.

Fig 1. Binary classification (binder vs. non-binder) performance.

(a) AUC of MHC-I binding epitope prediction tools. (b) ROC curves. IC50 = 500 nM was used as the cutoff for classifying experimentally measured epitopes. AUC was shown by box plot with upper and lower boundaries covering confidence level of 95%. (c) ROC curves enlarged for TPR between 0.7 and 1.0.

Fig 2. Evaluation of mhcflurry_pan predictor.

Comparison of prediction power, indicated by (a) AUC, (b) specificity of binders, and (c) specificity of strong binders, of 9mer-based mhcflurry with 9mer-based NetMHCpan4, 43mer-based testing HLA included (mhcflurry_pan) and testing HLA leave-one-out (mhcflurry_pan_LOO) pan-predictor. Each point represents one HLA type.

Fig 3

Multiclass classification performance of MHC class I binding epitope prediction tools: (a) VUS; (b) SPE; (c) SRCC; (d): R-squared of linear regression. IC50 thresholds of 50 nM and 500 nM were used to classifying experimental measurements between strong binder, weak binder, and non-binder. The box plots of VUS and SRCC show values covering 95% confidence level. Note that IC50 is not calculated in MixMHCpred.

Fig 4. Comparison of prediction accuracy between 9-mer and 10-mer testing data.

(a) ROC curves of 10-mer predictions with AUC value shown after each method. (b) Boxplots of AUC and SRCC calculated for 9-mer and 10-mer predictions, with each point representing a type I HLA allele. Significant levels were obtained by Wilcoxon test (*: p < 0.05; ns: p > 0.05).

Fig 5. Binary classification performance of MHC-II binding epitope prediction tools.

(a) AUC. (b) ROC curves. IC50 = 1000 nM was used as the cutoff for classifying experimentally measured epitopes. AUC was shown by box plot with upper and lower boundaries covering confidence level of 95%. (c) ROC curves enlarged for TPR between 0.7 and 1.0.

Fig 6

Multiclass classification performance of MHC-II binding epitope prediction tools: (a) VUS; (b) SPE; (c) SRCC; (d): R-squared of linear regression. The box plots of VUS and SRCC show values covering 95% confidence level.

Fig 7

Reliability of predicting absolute affinities of strong binding MHC Class I and II epitopes for (a) NetMHC4, (b) mhcflurry-class I, (c) nn_align, and (d) mhcflurry-class II. Measurement and prediction values were represented as 1-log10(IC50)/log10(50000 nM) and were light-colored based on 2-D data density. Grey dotted lines mark 50 nM threshold (y = 0.638) and grey dashed lines mark 500 nM (y = 0.426) threshold. Red lines show the linear regression of the data. FNr(50 nM) indicates the false negative rate of classifying strong binders.

Fig 8. Assessing the reliability of binding prediction methods for the identification of naturally processed MHC-epitopes.

(a) Box plots showing the quartile distribution of binding affinity rankings as predicted by NetMHC4 and NetMHCpan4 for MHC-eluted and non-eluted peptides. Grey dashed line indicates predicted binder ranking of top 2%. (b) FDr and FNr values calculated based on top 2% percentile rank cutoff on three MS-derived datasets visualized as heatmap. Similar plots of predicted scores are shown in S4 Fig.

Fig 9. Prediction of MHC class I epitopes by FlexPepDock.

(a) ROC curves and AUC values (shown after allele legend) generated based on reweighted binding energy scores reported by FlexPepDock. Peptides were labeled as positive or negative class by the IC50 = 500 nM cutoff. True positive and false positive were then calculated by correlating with FlexPepDock reweighted score. (b) and (c) Lowest energy conformations of 9-mers FLGGTPVCL and FLSHDFTLV to HLA-A0201 protein. MHC proteins are shown by orange ribbon and white surface; peptide backbones are shown in cyan; MHC-binding residues are shown in silver; potential T cell receptor contacting residues are shown in pink.