Comparison and integration of deleteriousness prediction methods for nonsynonymous SNVs in whole exome sequencing studies

Abstract

Accurate deleteriousness prediction for nonsynonymous variants is crucial for distinguishing pathogenic mutations from background polymorphisms in whole exome sequencing (WES) studies. Although many deleteriousness prediction methods have been developed, their prediction results are sometimes inconsistent with each other and their relative merits are still unclear in practical applications. To address these issues, we comprehensively evaluated the predictive performance of 18 current deleteriousness-scoring methods, including 11 function prediction scores (PolyPhen-2, SIFT, MutationTaster, Mutation Assessor, FATHMM, LRT, PANTHER, PhD-SNP, SNAP, SNPs&GO and MutPred), 3 conservation scores (GERP++, SiPhy and PhyloP) and 4 ensemble scores (CADD, PON-P, KGGSeq and CONDEL). We found that FATHMM and KGGSeq had the highest discriminative power among independent scores and ensemble scores, respectively. Moreover, to ensure unbiased performance evaluation of these prediction scores, we manually collected three distinct testing datasets, on which no current prediction scores were tuned. In addition, we developed two new ensemble scores that integrate nine independent scores and allele frequency. Our scores achieved the highest discriminative power compared with all the deleteriousness prediction scores tested and showed low false-positive prediction rate for benign yet rare nonsynonymous variants, which demonstrated the value of combining information from multiple orthologous approaches. Finally, to facilitate variant prioritization in WES studies, we have pre-computed our ensemble scores for 87 347 044 possible variants in the whole-exome and made them publicly available through the ANNOVAR software and the dbNSFP database.

Figure 1.

ROC curves for existing prediction scores and our ensemble scores. These two plots illustrated performance of quantitative prediction outcomes for existing prediction scores and our ensemble prediction scores evaluated by the ROC curve and AUC score for the ROC curve. Higher AUC score indicates better performance. Top plot used testing dataset I as benchmark dataset and bottom plot used testing dataset II as benchmark dataset (see Table 1). 95% CI indicates 95% confidence interval computed with 2000 stratified bootstrap replicates.

Figure 2.

Sensitivity and specificity plots for existing prediction scores and our ensemble scores. These two plots illustrated the performance of qualitative prediction outcomes of existing prediction scores and our ensemble prediction scores, evaluated by sensitivity and specificity. Higher sensitivity/specificity score indicates better performance. Top plot used testing dataset I as benchmark dataset and bottom plot used testing dataset II as benchmark dataset (see Table 1). The tegend table showed various qualitative prediction performance measurements for each prediction tool. MCC is a correlation coefficient between the observed and predicted binary classification, ranging from −1 to 1, where 1 indicates perfect prediction, −1 indicates total disagreement between prediction and observation. $\text{MCC}=\left((\text{TP}\times \text{TN}-\text{FP}\times \text{FN})/\left(\sqrt{(\text{TP + FP})(\text{TP + FN})(\text{TN + FP})(\text{TN + FN})}\right)\right),$ where TP, TN, FP and FN denotes true positive, true negative, false positive and false negative, respectively. ACC denotes accuracy; $\text{ACC}=((\mathrm{TP}+\text{TN)}/(\mathrm{TP}+\text{FP}+\text{TN}+\text{FN))}$. TPR denotes true positive rate, or sensitivity; $\text{TPR}=(\mathrm{TP}/(\text{TP}+\text{FN}))$. TNR denotes true negative rate, or specificity; $\text{TNR}=(\mathrm{TN}/(\text{TN}+\text{FP}))$. FPR denotes false-positive rate; $\text{FPR}=(\text{FP}/(\text{TN}+\text{FP}))$. FNR denotes false-negative rate; $\text{FNR}=(\text{FN}/(\text{TP}+\text{FN}))$. PPV denotes positive predictive value; $\text{PPV}=(\text{TP}/(\text{TP}+\text{FP}))$. NPV denotes negative predictive value; $\text{NPV}=(\text{TN}/(\text{TN}+\text{FN}))$. FDR denotes false discovery rate; $\text{FDR}=(\text{FP}/(\text{FP}+\text{TP}))$. For each qualitative prediction performance measurement, top three performance scores were highlighted. The brighter the highlight color, the better the performance.