Evaluating the Quality of Studies Assessing COVID-19 Vaccine Neutralizing Antibody Immunogenicity

Abstract

Objective: COVID-19 vaccine-neutralizing antibodies provide early data on potential vaccine effectiveness, but their usefulness depends on study reliability and reporting quality. Methods: We systematically evaluated 50 published post-vaccination neutralizing antibody studies for key parameters that determine study and data quality regarding sample size, SARS-CoV-2 infection, vaccination regimen, sample collection period, demographic characterization, clinical characterization, experimental protocol, live virus and pseudo-virus details, assay standardization, and data reporting. Each category was scored from very high to low or unclear quality, with the lowest score determining the overall study quality score. Results: None of the studies attained an overall high or very high score, 8% (n = 4) attained moderate, 42% (n = 21) low, and 50% (n = 25) unclear. The categories with the fewest studies assessed as ≥ high quality were SARS-CoV-2 infection (42%), sample size (30%), and assay standardization (14%). Overall quality was similar over time. No association between journal impact factor and quality score was found. Conclusions: We found that reporting in neutralization studies is widely incomplete, limiting their usefulness for downstream analyses.

Keywords: COVD-19; SARS-CoV-2; neutralizing antibodies; reliability; reporting quality.

Figure 1

Individual quality scores for each category of all assessed studies. Studies are sorted for impact factor and publication year. References [17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66] to the respective study number is provided in Supplementary Table S1. Quality scores from low to very high are provided for eleven categories. The maximum impact is “unclear” for the categories of vaccination regimen, protocol, live virus, pseudo-virus, and data reporting; all remaining categories have “low” as the lowest quality score. Overall quality is assigned the lowest score observed across categories. For each study, the percentage of the 11 categories with high or very high quality scores and unclear scores is provided. ¹ Unclear quality is specified as no sufficient information to evaluate the quality of the data, indicating low reporting quality.

Figure 2

Percentage of studies receiving specific quality scores for each category of the QAT. Quality ratings from low to very high quality scores on the x-axis. If the maximum possible impact of a category is unclear, low quality is not applicable (n.a.) for that category. The percentage of studies with the respective quality scores is provided above the bars.

Figure 3

Comparison of quality scores for each category of studies published in 2021 vs. 2023. The percentage of studies with very high, high, moderate, unclear, and low quality scores are shown by publication year (February–May 2021 and March–July 2023) for 25 studies in each period. p-values for comparisons of very high or high vs. other categories are provided for each category. ^x The maximum impact is “unclear” for the categories of vaccination regimen, protocol, live virus, pseudo-virus, and data reporting; all remaining categories have “low” as the maximum quality score. Overall quality is assigned the lowest score observed across categories. ¹ Unclear quality is specified as no sufficient information to evaluate the quality of the data, indicating low reporting quality. Statistical analysis was performed by Student’s t-test. Statistical significance was defined by a value of * <0.05; ** <0.01; ns, not significant.