Virus sequencing performance during the SARS-CoV-2 pandemic: a retrospective analysis of data from multiple rounds of external quality assessment in Austria

Abstract

Introduction: A notable feature of the 2019 coronavirus disease (COVID-19) pandemic was the widespread use of whole genome sequencing (WGS) to monitor severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections. Countries around the world relied on sequencing and other forms of variant detection to perform contact tracing and monitor changes in the virus genome, in the hopes that epidemic waves caused by variants would be detected and managed earlier. As sequencing was encouraged and rewarded by the government in Austria, but represented a new technicque for many laboratories, we designed an external quality assessment (EQA) scheme to monitor the accuracy of WGS and assist laboratories in validating their methods. Methods: We implemented SARS-CoV-2 WGS EQAs in Austria and report the results from 7 participants over 5 rounds from February 2021 until June 2023. The participants received sample material, sequenced genomes with routine methods, and provided the sequences as well as information about mutations and lineages. Participants were evaluated on the completeness and accuracy of the submitted sequence and the ability to analyze and interpret sequencing data. Results: The results indicate that performance was excellent with few exceptions, and these exceptions showed improvement over time. We extend our findings to infer that most publicly available sequences are accurate within ≤1 nucleotide, somewhat randomly distributed through the genome. Conclusion: WGS continues to be used for SARS-CoV-2 surveillance, and will likely be instrumental in future outbreak scenarios. We identified hurdles in building next-generation sequencing capacity in diagnostic laboratories. EQAs will help individual laboratories maintain high quality next-generation sequencing output, and strengthen variant monitoring and molecular epidemiology efforts.

Keywords: Austria; COVID-19 diagnostic testing; SARS-CoV-2; diagnostic laboratories; next-generation sequencing.

FIGURE 1

Genome completeness (A) and percent of samples undetected (B) over five rounds of a SARS-CoV-2 sequencing EQA in Austria. Points for genome completeness (A) are shown over approximate sample concentration (estimated by C _t value) from each submitted result (n = 54). The size of points in percent of results undetected (B) is relative to the number of participants (between 3 and 7) that submitted a result for a given sample (n = 22). In both panels the points are transparent gray and appear darker when overlapping.

FIGURE 2

Sequence accuracy scores for SARS-CoV-2 sequencing EQAs (A) and scoring for self-reported mutation in the spike gene (B). Each point represents the result from each sample submitted by each participant for each of five rounds of EQAs with boxes around the interquartile range and a thick horizontal line indicating the median value. The sequence accuracy score (A) is the number of differences from the consensus sequence of all laboratories for each sample. The accuracy score was independently verified for the 3^rd, 4^th, and 5^th rounds, but were inferred from submitted mutations for the 1^st and 2^nd rounds as no fastn files were requested during those two rounds. The self-reported mutation score (B) is a count of the differences between the non-synonymous amino acid mutations in the spike protein reported by the participant and an independent analysis of the mutations by the evaluator, which was calculated for the 3^rd, 4^th, and 5^th rounds. In both panels the points are transparent gray and appear darker when overlapping.

FIGURE 3

A genomic map of all whole genome sequences submitted over three rounds of a SARS-CoV-2 sequencing EQA in Austria missing data are thick black bands. Differences from the consensus and their approximate location on the genome are indicated by colored shapes: red squares indicate ambiguous nucleotides were called; orange circles represent indels that created a frameshift within an open reading frame; yellow triangles indicate a “private” difference from the consensus sequence that was not in the reference sequence (Wuhan-Hu-1, NC_045512); and green diamonds indicate mutation in the consensus sequence relative to the reference sequence (Wuhan-Hu1) that was not detected (“absent”). The sequence maps are grouped and labeled by round of EQA, laboratory letter as in Table 2, and Sample Number (Round_Lab_Sample) on the left. Letters on the right indicate results from duplicated samples (“D”) or the sample with minor variants (“M”).

FIGURE 4

Summary of average completeness, accuracy score, and mutation score of each participant in each round. The colored tiles are shaded according to a gradient as indicated in each legend. White indicates no participation in that round. Gray squares indicate participation but data were missing. Specifically, accuracy scores could be interpreted from submitted data in rounds 1 and 2, but were not independently verifiable as no fastn sequences were submitted. Completeness scores from two laboratories in round 1 were based on submitted genome completeness information that was not specifically requested.