Performance of in silico tools for the evaluation of p16INK4a (CDKN2A) variants in CAGI

Abstract

Correct phenotypic interpretation of variants of unknown significance for cancer-associated genes is a diagnostic challenge as genetic screenings gain in popularity in the next-generation sequencing era. The Critical Assessment of Genome Interpretation (CAGI) experiment aims to test and define the state of the art of genotype-phenotype interpretation. Here, we present the assessment of the CAGI p16INK4a challenge. Participants were asked to predict the effect on cellular proliferation of 10 variants for the p16INK4a tumor suppressor, a cyclin-dependent kinase inhibitor encoded by the CDKN2A gene. Twenty-two pathogenicity predictors were assessed with a variety of accuracy measures for reliability in a medical context. Different assessment measures were combined in an overall ranking to provide more robust results. The R scripts used for assessment are publicly available from a GitHub repository for future use in similar assessment exercises. Despite a limited test-set size, our findings show a variety of results, with some methods performing significantly better. Methods combining different strategies frequently outperform simpler approaches. The best predictor, Yang&Zhou lab, uses a machine learning method combining an empirical energy function measuring protein stability with an evolutionary conservation term. The p16INK4a challenge highlights how subtle structural effects can neutralize otherwise deleterious variants.

Keywords: CAGI experiment; bioinformatics tools; cancer; pathogenicity predictors; variant interpretation.

Figure 1. Overview of CDK6-P16INK4A tumor suppressor complex

Cartoon representations of the p16INK4a 3D structure (PDB code 1BI7) colored blue, while CDK6 is presented as full surface (light grey). Magenta spheres represent positions of variants considered for the challenge mapped on its surface. The ankyrin repeats composing p16INK4a structure are presented below with a schematic representation of mutated amino acid positions (magenta spots). Variant nomenclature refers to CDKN2A mRNA isoform1 (GenBank identifier: NM_000077.4), nucleotide numbering starts with the A of the ATG translation initiation site.

Figure 2. Correlation among submissions

Each cell shows the Pearson correlation coefficient between two submissions, with a color scale ranging from green (+1, perfect correlation) to red (0, no correlation) and black (−1, perfect anti-correlation). Submissions are clustered by group.

Figure 3. Correlation among performance indices

Each cell shows the Kendall correlation coefficient between the two corresponding measures, with a color scale ranging from green (+1, perfect correlation) to red (−1, perfect anti-correlation). Notice how similar measures tend to cluster together. The four selected measures are highlighted in bold face.

Figure 4. Pairwise difference between submissions

Statistical differences between submissions based on the overall ranking achieved by each submission, sorted according to the final ranking. White squares are indices of tied predictions (P-values > 0.05) meaning that performances are similar and the difference between two predictors is not statistically significant.