Development of an evidence-based algorithm that optimizes sensitivity and specificity in ES-based diagnostics of a clinically heterogeneous patient population

Abstract

Purpose: Next-generation sequencing (NGS) is rapidly replacing Sanger sequencing in genetic diagnostics. Sensitivity and specificity of NGS approaches are not well-defined, but can be estimated from applying NGS and Sanger sequencing in parallel. Utilizing this strategy, we aimed at optimizing exome sequencing (ES)-based diagnostics of a clinically diverse patient population.

Methods: Consecutive DNA samples from unrelated patients with suspected genetic disease were exome-sequenced; comparatively nonstringent criteria were applied in variant calling. One thousand forty-eight variants in genes compatible with the clinical diagnosis were followed up by Sanger sequencing. Based on a set of variant-specific features, predictors for true positives and true negatives were developed.

Results: Sanger sequencing confirmed 81.9% of ES-derived variants. Calls from the lower end of stringency accounted for the majority of the false positives, but also contained ~5% of the true positives. A predictor incorporating three variant-specific features classified 91.7% of variants with 100% specificity and 99.75% sensitivity. Confirmation status of the remaining variants (8.3%) was not predictable.

Conclusions: Criteria for variant calling in ES-based diagnostics impact on specificity and sensitivity. Confirmatory sequencing for a proportion of variants, therefore, remains a necessity. Our study exemplifies how these variants can be defined on an empirical basis.

Keywords: Genetic testing; Laboratory standards; Sensitivity; Specificity; exome sequencing.

Fig. 1. Performance of filtering criteria commonly used during variant calling in next-generation sequencing (NGS).

a Distribution of Sanger-confirmed vs. Sanger-rejected variants in our data set. b Modeling of the effects of standard panel- and exome sequencing (ES)-associated filtering cutoffs. c Contribution of individual parameters to the drop in sensitivity (stippled and dotted gray lines: common cutoffs for panel-based and ES-based approaches, respectively; solid gray lines: cutoffs initially used by the present study). The asterisk denotes that sensitivity as indicated here is not equivalent to the (unknown) overall sensitivity of the assay, but refers to our set of 1048 Sanger followed-up candidate variants

Fig. 2. Features of exome sequencing (ES)-derived candidate variants, and association with Sanger confirmation status.

Stippled gray lines indicate the corresponding values for all 1,048 variants. a Analogous features. Significance of differences between means (+/- SEM) was calculated using the two-sided Student’s t-test. b Digital features. Significance of differences was calculated using Fisher’s exact test

Fig. 3. Iterative analyses aiming at the definition of classifiers that can predict true positive or false positive ES-derived candidate variants while maintaining high specificity and sensitivity.

Analogous features are depicted as receiver operating characteristic curves. Binary classifications according to digital features are indicated by filled symbols. a First round of analysis on all 1,048 variants. Arrow: “quality”-based binary classifier that correctly predicts status “confirmed” for 813 variants; 100% specificity is thus retained. b Second round of analysis on the 235 variants that remained after round 1. Arrow: “variant reads”–based binary classifier that correctly predicts status “not confirmed” for 87 of 88 variants. Overall sensitivity decreases to 99.9%. c Third round of analysis on the 147 variants that remained after round 2. Arrow: “frequency”-based binary classifier that correctly predicts status “not confirmed” for 59 of 60 variants. Overall sensitivity decreases to 99.8%. d A fourth round of analysis on the 87 variants remaining after round 3 does not define additional classifiers

Fig. 4. Characteristics of the three groups created by our classifiers.

a Group-specific means (+/- SEM) for analogous features (upper panel), and intragroup distribution of states for digital features (lower panel). Note that for the majority of features, the variants that require Sanger sequencing occupy an intermediate position. b Performance of filtering criteria in the set of variants that require Sanger-based confirmation (compare Fig. 1a, b)

Fig. 5. Workflow for following up on identification of a candidate variant by exome sequencing (ES).

Note that this workflow is based on the data set presented in the present study, and is also specific to it (corresponding numbers of variants are indicated in parentheses). Gray sectors and corresponding percentages in pie charts indicate the remaining Sanger sequencing load