Meta-analyses of diagnostic test accuracy could not be reproduced

Abstract

Background and objectives: The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews.

Study design and setting: We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through Ovid. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. Successful reproduction was defined as a result differing <1% point from the reported point estimates; or reported primary study results that were in line with those of the actual primary study results; or if the data from the primary studies could be extracted without checking the data in the review first.

Results: Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2×2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced. Considering the correctness of the numbers from the primary articles and the complete reporting of the search strategy, only one meta-analysis was fully replicable.

Conclusion: Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used, and partly because of mistakes in the data extraction or data reporting.

Keywords: Diagnostic test accuracy; Meta-analyses; Reproducibility; Sensitivity and specificity; Systematic reviews; Transparency.