Performance of in silico prediction tools for the classification of rare BRCA1/2 missense variants in clinical diagnostics

Abstract

Background: The use of next-generation sequencing approaches in clinical diagnostics has led to a tremendous increase in data and a vast number of variants of uncertain significance that require interpretation. Therefore, prediction of the effects of missense mutations using in silico tools has become a frequently used approach. Aim of this study was to assess the reliability of in silico prediction as a basis for clinical decision making in the context of hereditary breast and/or ovarian cancer.

Methods: We tested the performance of four prediction tools (Align-GVGD, SIFT, PolyPhen-2, MutationTaster2) using a set of 236 BRCA1/2 missense variants that had previously been classified by expert committees. However, a major pitfall in the creation of a reliable evaluation set for our purpose is the generally accepted classification of BRCA1/2 missense variants using the multifactorial likelihood model, which is partially based on Align-GVGD results. To overcome this drawback we identified 161 variants whose classification is independent of any previous in silico prediction. In addition to the performance as stand-alone tools we examined the sensitivity, specificity, accuracy and Matthews correlation coefficient (MCC) of combined approaches.

Results: PolyPhen-2 achieved the lowest sensitivity (0.67), specificity (0.67), accuracy (0.67) and MCC (0.39). Align-GVGD achieved the highest values of specificity (0.92), accuracy (0.92) and MCC (0.73), but was outperformed regarding its sensitivity (0.90) by SIFT (1.00) and MutationTaster2 (1.00). All tools suffered from poor specificities, resulting in an unacceptable proportion of false positive results in a clinical setting. This shortcoming could not be bypassed by combination of these tools. In the best case scenario, 138 families would be affected by the misclassification of neutral variants within the cohort of patients of the German Consortium for Hereditary Breast and Ovarian Cancer.

Conclusion: We show that due to low specificities state-of-the-art in silico prediction tools are not suitable to predict pathogenicity of variants of uncertain significance in BRCA1/2. Thus, clinical consequences should never be based solely on in silico forecasts. However, our data suggests that SIFT and MutationTaster2 could be suitable to predict benignity, as both tools did not result in false negative predictions in our analysis.

Keywords: BRCA; Classification; Missense variant; Prediction tools; Variant of uncertain significance.

Fig. 1

Multifactorial probability model and determination of Product of LRs thresholds for classification irrespective of Align-GVGD. Visualization of the multifactorial probability model for classification of VUS in BRCA1/2 based on the Posterior P thresholds proposed by Plon et al. [22]. Variants with Products of LRs below or above the corresponding thresholds (indicated with filled circles) were classified independent of the prior probabilities Prior P (based on Align-GVGD predictions) and included in the Evaluation Variant Set and the Classified Variant Set. Variants depicted by unfilled circles were included in the Classified Variant Set exclusively. Classification of these variants was not independent of prior probabilities

Fig. 2

Performance of in silico prediction tools as stand-alone methods or in combination. Sensitivity (SENS), specificity (SPEC), accuracy (ACC) and Matthews correlation coefficient (MCC) of stand-alone tools and combinations of prediction tools Align-GVGD, SIFT, MutationTaster2 (MT) and PolyPhen-2 (PPhen-2) as observed and estimated from the sensitivities and specificities of stand-alone methods on the Evaluation Variant Set of 166 missense variants on BRCA1 and BRCA2. Align-GVGD, SIFT and MutationTaster2 reached values for sensitivity > 0.92 as stand-alone tools as well as in combination. The comparatively low sensitivity of PolyPhen-2 as a stand-alone approach is also reflected in the decreased sensitivities of combined approaches involving PolyPhen-2. The specificities of stand-alone tools varied between 0.67 (PolyPhen-2) and 0.92 (Align-GVGD), and the specificities of combined approaches increased with increasing m. False negatives (false positives, respectively) denote the number of index patients tested in GC-HBOC as of September 2016 that would receive an erroneous negative (respectively positive) result if the diagnosis were based solely on the corresponding in silico approach

Fig. 3

Sensitivities and specificities of the stand-alone prediction tools Align-GVGD, SIFT, MutationTaster2 (MT) and PolyPhen-2 (PPhen-2). The sensitivities and specificities varied between the tools and with values of ≥0.90, Align GVGD performed best for the Evaluation Data Set. Performance for BRCA1 and BRCA2 variants was comparable for all tools, except for PolyPhen-2 which showed a lower sensitivity of only 0.56 on BRCA1 variants compared with 1.0 on BRCA2 variants and a specificity of 0.67 on BRCA1 compared with 0.72 on BRCA2 variants. Comparison between the Classified Variant Set and Evaluation Variant Set revealed only minor differences in sensitivity and specificity for the four tools examined here

Fig. 4

Venn diagrams summarizing the in silico prediction on 670 VUS from the GC-HBOC database. Variants classified as pathogenic by at least one program out of Align-GVGD, SIFT, MutationTaster2 and PolyPhen-2 are depicted in a), and variants classified as benign are shown in b). A total of 354 VUS were consistently classified as benign by all four tools under consideration, and 57 variants were consistently classified as pathogenic. In contrast, 57 variants were classified as benign exclusively by Align-GVGD, whereas 68 (30) were classified as pathogenic exclusively by PolyPhen-2 (SIFT). These inconsistent predictions point toward a noticeable amount of misclassifications by the corresponding tool