Evaluation of in silico algorithms for use with ACMG/AMP clinical variant interpretation guidelines

Abstract

Background: The American College of Medical Genetics and American College of Pathologists (ACMG/AMP) variant classification guidelines for clinical reporting are widely used in diagnostic laboratories for variant interpretation. The ACMG/AMP guidelines recommend complete concordance of predictions among all in silico algorithms used without specifying the number or types of algorithms. The subjective nature of this recommendation contributes to discordance of variant classification among clinical laboratories and prevents definitive classification of variants.

Results: Using 14,819 benign or pathogenic missense variants from the ClinVar database, we compared performance of 25 algorithms across datasets differing in distinct biological and technical variables. There was wide variability in concordance among different combinations of algorithms with particularly low concordance for benign variants. We also identify a previously unreported source of error in variant interpretation (false concordance) where concordant in silico predictions are opposite to the evidence provided by other sources. We identified recently developed algorithms with high predictive power and robust to variables such as disease mechanism, gene constraint, and mode of inheritance, although poorer performing algorithms are more frequently used based on review of the clinical genetics literature (2011-2017).

Conclusions: Our analyses identify algorithms with high performance characteristics independent of underlying disease mechanisms. We describe combinations of algorithms with increased concordance that should improve in silico algorithm usage during assessment of clinically relevant variants using the ACMG/AMP guidelines.

Keywords: ACMG; ClinVar; Clinical genetics; Diagnostics; In silico algorithm; ROC; Variant interpretation.

Fig. 1

Concordance among predictions of 18 algorithms for variants in ClinVar. Binary predictions made by 18 algorithms for each pathogenic or benign variants in ClinVar are shown in the upper and lower panels. Each variant is along a row and an orange, green, or white tile depicts a pathogenic, benign, or missing data call, respectively, by the corresponding algorithm. A total of 14,819 variants with ClinVar review status one star or above (a) and 2966 variants with ClinVar review status two stars or above (b) are shown

Fig. 2

Concordance among algorithms. a Distribution of proportion of variants that had concordant calls by any given pair of algorithms (among 18 algorithms) for benign (green) and pathogenic (orange) variants in ClinVar. b Scatterplots of true concordance (variant assertion matches ClinVar assertion) vs false concordance (variant assertion does not match ClinVar assertion) for combinations of three, four, or five algorithms at a time. An orange and a green point depict the true and false concordance of a combination for benign and pathogenic variants, respectively, in ClinVar. The rugs on top and bottom, left and right represent the distribution of false and true concordances, respectively. c Hierarchical clustering of 25 algorithms with scores for 14,819 variants in ClinVar. Red rectangles indicate robust clusters with an AU p value of > 0.99 (see “Methods”)

Fig. 3

Performance analysis of algorithms. The AUC of a ROC are plotted for 25 algorithms. Vertical dotted line indicates an AUC of 0.9 and 99% confidence intervals for each AUC are shown. Blue dots indicate AUC > 0.89. a AUCs of the algorithms across different datasets shown in the panels and described in text. b AUCs of the algorithms across different datasets (represented in panels) to address type I circularity as described in text. The same plots for ClinVar Status * and ClinVar Status ** as in Fig. 3a are used in 3b for comparison. Any instance of ** represents variants with ClinVar review status of two stars or above.   Ensemble predictors are indicated by dark green labels on the y-axis