Application of weighting methods for presenting risk-of-bias assessments in systematic reviews of diagnostic test accuracy studies

Abstract

Background: An assessment of the validity of individual diagnostic accuracy studies in systematic reviews is necessary to guide the analysis and the interpretation of results. Such an assessment is performed for each included study and typically reported at the study level. As studies may differ in sample size and disease prevalence, with larger studies contributing more to the meta-analysis, such a study-level report does not always reflect the risk of bias in the total body of evidence. We aimed to develop improved methods of presenting the risk of bias in the available evidence on diagnostic accuracy of medical tests in systematic reviews, reflecting the relative contribution of the study to the body of evidence in the review.

Methods: We applied alternative methods to represent evaluations with the Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2), weighting studies according to their relative contribution to the total sample size or their relative effective sample size. We used these methods in four existing systematic reviews of diagnostic accuracy studies, including 9, 13, 22, and 32 studies, respectively.

Results: The risk-of-bias summaries for each domain of the QUADAS-2 checklist changed in all four sets of studies after replacing unit weights for the studies with relative sample sizes or with the relative effective sample size. As an example, the risk of bias was high in the patient selection domain in 31% of the studies in one review, unclear in 23% and low in 46% of studies. Weighting studies according to the relative sample size changed the corresponding proportions to 4%, 4%, and 92%, respectively. The difference between the two weighting methods was small and more noticeable when the reviews included a smaller number of studies with wider range of sample size.

Conclusions: We present an alternative way of presenting the results of risk-of-bias assessments in systematic reviews of diagnostic accuracy studies. Weighting studies according to their relative sample size or their relative effective sample size can be used as more informative summaries of the risk of bias in the total body of available evidence.

Systematic review registrations: Not applicable.

Keywords: Diagnostic accuracy studies; Quality appraisal; Risk-of-bias assessment; Systematic reviews.

Fig. 1

Results of risk-of-bias assessment plots, which illustrate the judgments (“Low risk,” “High risk,” and “Unclear risk” of bias) for four QUADAS-2 tool domains (x-axis) based on (A) proportion of included studies, (B) proportion of included patients (C) effective sample size of 13 included studies in the ELF systematic review (y-axis)

Fig. 2

Results of risk-of-bias assessment plots, which illustrate the judgments (“Low risk,” “High risk,” and “Unclear risk” of bias) for four QUADAS-2 tool domains (x-axis) based on (A) proportion of included studies, (B) proportion of included patients (C) effective sample size of 32 included studies in the CK18 systematic review (y-axis)

Fig. 3

Results of risk-of-bias assessment plots, which illustrate the judgments (“Low risk,” “High risk,” and “Unclear risk” of bias) for four QUADAS-2 tool domains (x-axis) based on (A) proportion of included studies, (B) proportion of included patients (C) effective sample size of 22 included studies in the Lombardi 2020 systematic review (y-axis)

Fig. 4

Results of risk-of-bias assessment plots, which illustrate the judgments (“Low risk,” “High risk,” and “Unclear risk” of bias) for four QUADAS-2 tool domains (x-axis) based on (A) proportion of included studies, (B) proportion of included patients (C) effective sample size of 9 included studies in the Mattioni 2020 systematic review (y-axis)