Calibration of additional computational tools expands ClinGen recommendation options for variant classification with PP3/BP4 criteria

Abstract

Purpose: We previously developed an approach to calibrate computational tools for clinical variant classification, updating recommendations for the reliable use of variant impact predictors to provide evidence strength up to Strong. A new generation of tools using distinctive approaches have since been released, and these methods must be independently calibrated for clinical application.

Method: Using our local posterior probability-based calibration and our established data set of ClinVar pathogenic and benign variants, we determined the strength of evidence provided by three new tools (AlphaMissense, ESM1b, VARITY) and calibrated scores meeting each evidence strength.

Results: All three tools reached the Strong level of evidence for variant pathogenicity and Moderate for benignity, though sometimes for few variants. Compared to previously recommended tools, these yielded at best only modest improvements in the tradeoffs of evidence strength and false positive predictions.

Conclusion: At calibrated thresholds, three new computational predictors provided evidence for variant pathogenicity at similar strength to the four previously recommended predictors (and comparable with functional assays for some variants). This calibration broadens the scope of computational tools for application in clinical variant classification. Their new approaches offer promise for future advancement of the field.

Keywords: ACMG/AMP; AlphaMissense; ESM1b; PP3/BP4; VARITY; calibration; clinical variant classification.

Figure 1.. Local posterior probability curves and comparison with previously calibrated tools.

(A) Pairs of curves for AlphaMissense, ESM1b and VARITY_R. For each tool, the curve on the left is for pathogenicity (red horizontal lines) and the curve on the right is for benignity (blue horizontal lines). The horizontal lines represent the posterior probability thresholds for Supporting, Moderate, Strong, and Very strong evidence as per current ACMG/AMP guidelines. A horizontal line representing the 3-point strength of evidence is also shown. The black curves represent the posterior probability estimated from the ClinVar 2019 set. The gray curves represent one-sided 95% confidence intervals (in the direction of more stringent thresholds), calculated from 10,000 bootstrap samples of this data set. The points at which the gray curves intersect the horizontal lines represent the thresholds for the relevant intervals. (B) The likelihood ratios within each interval on the independent ClinVar 2020 set. Darker colors indicate higher values for pathogenicity and lower values for benignity (because these are positive likelihood ratios). The limits for the color gradients are asymmetric, with ranges set between zero and one for benignity, and one and 100 for pathogenicity. A gray rectangle is introduced at the center for comparability with (C). (C) The percentage of variants predicted to be within the interval in the gnomAD set. Blue and red distinguish the evidential strength intervals for benignity from pathogenicity, respectively, with the indeterminate interval colored gray. The color gradient corresponds to the value in the cells, regardless of color. Darker colors indicate higher proportions. A white cell without a value indicates that the tool did not reach thresholds corresponding to that interval. The indeterminate interval also included variants without any scores.