Assessing predictions on fitness effects of missense variants in HMBS in CAGI6

Abstract

This paper presents an evaluation of predictions submitted for the "HMBS" challenge, a component of the sixth round of the Critical Assessment of Genome Interpretation held in 2021. The challenge required participants to predict the effects of missense variants of the human HMBS gene on yeast growth. The HMBS enzyme, critical for the biosynthesis of heme in eukaryotic cells, is highly conserved among eukaryotes. Despite the application of a variety of algorithms and methods, the performance of predictors was relatively similar, with Kendall's tau correlation coefficients between predictions and experimental scores around 0.3 for a majority of submissions. Notably, the median correlation (≥ 0.34) observed among these predictors, especially the top predictions from different groups, was greater than the correlation observed between their predictions and the actual experimental results. Most predictors were moderately successful in distinguishing between deleterious and benign variants, as evidenced by an area under the receiver operating characteristic (ROC) curve (AUC) of approximately 0.7 respectively. Compared with the recent two rounds of CAGI competitions, we noticed more predictors outperformed the baseline predictor, which is solely based on the amino acid frequencies. Nevertheless, the overall accuracy of predictions is still far short of positive control, which is derived from experimental scores, indicating the necessity for considerable improvements in the field. The most inaccurately predicted variants in this round were associated with the insertion loop, which is absent in many orthologs, suggesting the predictors still heavily rely on the information from multiple sequence alignment.

Fig 1.. Distributions of experimental fitness scores and predicted scores.

(A) Histogram showing the distribution of experimental fitness scores for nonsense and synonymous mutations (left) and missense mutations (right); (B) histograms of predicted scores from a selected submission from each participating team. The Y-axis represents the proportion of mutations, while the X-axis represents experimental scores in panels (A) and (B)

Fig 2.. Performance assessment of predictors.

(A) Receiver Operating Characteristic (ROC) curves for predicting deleterious mutations; (B) Head-to-head comparison matrix of predictors, with colors indicating the number of datasets in which one predictor (row) outperforms another (column); (C) Boxplot of the distribution of ranks for predictors in simulated datasets. The box edges represent the first and third quartiles of the ranks, the line inside the box denotes the median rank, whiskers extend to 1.5 times the interquartile range from the box edges, and circles represent outliers beyond 1.5 times the interquartile range.

Fig 3.. Effects of mutations on functional loops were poorly predicted by top-performing predictors.

(A) Heatmap of the median differences between experimental scores and those of the top-performing predictors at each position, with blue indicating lower and red indicating higher differences; (B) Structural representation of HEM3 (PDB ID: 5m6r, chain A) highlighting the active-site loop, cofactor-binding loop, insertion region, and residues 354 to 356 in red. ES2 and the phosphate group are displayed as spheres; (C) Distributions of experimental scores (blue) and predicted scores from submission 5_1 (green) within the active-site loop, cofactor-binding loop, and insertion region.

Fig 4.. Correlation among predictors and the role of conservation in prediction.

(A) A heatmap displaying absolute Kendall’s tau correlation coefficients between predictors. The absolute correlation coefficients are color-coded, with blue indicating lower and red indicating higher correlation; (B) Scatter plots depicting the correlation between the conservation index and the median of all predicted scores (left) or experimental scores (right) for mutations at each position. The Y-axis represents the median predicted/experimental score, while the X-axis represents the conservation index; (C) Bar graphs showing the ratio of deleterious mutations at conserved positions as indicated by experimental scores and predictors (upper graph) and the ratio of benign mutations at unconserved positions as indicated by experimental scores and predictors (lower graph).