CAGI6 ID panel challenge: assessment of phenotype and variant predictions in 415 children with neurodevelopmental disorders (NDDs)

Abstract

The Genetics of Neurodevelopmental Disorders Lab in Padua provided a new intellectual disability (ID) Panel challenge for computational methods to predict patient phenotypes and their causal variants in the context of the Critical Assessment of the Genome Interpretation, 6th edition (CAGI6). Eight research teams submitted a total of 30 models to predict phenotypes based on the sequences of 74 genes (VCF format) in 415 pediatric patients affected by Neurodevelopmental Disorders (NDDs). NDDs are clinically and genetically heterogeneous conditions, with onset in infant age. Here, we assess the ability and accuracy of computational methods to predict comorbid phenotypes based on clinical features described in each patient and their causal variants. We also evaluated predictions for possible genetic causes in patients without a clear genetic diagnosis. Like the previous ID Panel challenge in CAGI5, seven clinical features (ID, ASD, ataxia, epilepsy, microcephaly, macrocephaly, hypotonia), and variants (Pathogenic/Likely Pathogenic, Variants of Uncertain Significance and Risk Factors) were provided. The phenotypic traits and variant data of 150 patients from the CAGI5 ID Panel Challenge were provided as training set for predictors. The CAGI6 challenge confirms CAGI5 results that predicting phenotypes from gene panel data is highly challenging, with AUC values close to random, and no method able to predict relevant variants with both high accuracy and precision. However, a significant improvement is noted for the best method, with recall increasing from 66% to 82%. Several groups also successfully predicted difficult-to-detect variants, emphasizing the importance of variants initially excluded by the Padua NDD Lab.

Fig. 1

Summary of CAGI-6 ID panel challenge dataset. a The number of patients where the presence or absence of the phenotype was ascertained by a clinician. b For the 415 patients included in the study, the Padua NDD lab noted at least one variant relevant to the phenotype in 43.4% of the patients

Fig. 2

Overall performance for each submission on phenotype prediction. A Each cell represents MCC values. The color scale ranges from green (+ 1, perfect correlation) to red (− 1, negative correlation). White means no better than random prediction. B Each cell represents the mean AUC values of the ROC for 1000 bootstrap iterations. The color scale ranges from dark (+ 1, perfect performance) to white (0, random performance). C Standard deviation (SD) of the bootstrapped AUC values shown in B. AUC, area under ROC curve; MCC, Matthew correlation coefficient; ROC, receiver operating characteristic

Fig. 3

Distribution of the ROC curves for all seven clinical traits. The best performant submission for each phenotype, based on the AUC value, is shown

Fig. 4

Performance of the eight groups matching the specific phenotype in 415 patients. Colors represent the proportion and number of groups which correctly predicted the phenotype

Fig. 5

Predicted variants distribution. Category “Dataset” is the amount of variants which were identified and classified by the Padua NDD lab. Each bar represents the amount of variants and types predicted by each submission. NDD, neurodevelopmental disorder

Fig. 6

Performance of the eight groups predicting the correct variants. The amount of variants was calculated for each category (P/LP, VUS, RF). Colors indicate the proportion and number of groups which correctly predicted those variants