Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Feb 13;14(2):e075173.
doi: 10.1136/bmjopen-2023-075173.

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

Li Jiu  1 , Michiel Hartog  1 , Junfeng Wang  2 , Rick A Vreman  1 , Olaf H Klungel  1 , Aukje K Mantel-Teeuwisse  1 , Wim G Goettsch  2   3
Affiliations

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

Li Jiu et al. BMJ Open. .

Abstract

Objectives: We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

Design: Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

Data sources: Systematic search: Medline; Snowballing: starting from three articles (D'Andrea et al, Quigley et al and Faria et al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

Eligibility criteria for selecting studies: We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

Data extraction and synthesis: Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

Results: Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

Conclusion: Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

Osf registration number: OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).

Keywords: EPIDEMIOLOGY; HEALTH ECONOMICS; Systematic Review.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Figures

Figure 1
Figure 1
Flow chart for the inclusion and exclusion of appraisal tools for non-randomised studies of interventions
Figure 2
Figure 2
The extent to which the appraisal tools addressed quality items on methodological quality or reporting.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Makady A, de Boer A, Hillege H, et al. . What is real-world data? a review of definitions based on literature and stakeholder interviews. Value Health 2017;20:858–65. 10.1016/j.jval.2017.03.008 Available: mid: https://www.valueinhealthjournal.com/article/S1098-3015(17)30171-7/... - DOI - PubMed
    1. Reeves BC, Deeks JJ, Higgins JPT, et al. . Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions version 6. 2022: 3. Available: https://training.cochrane.org/handbook/current/chapter24#:~:text=NRSI%20...
    1. Higgins J, Morgan R, Rooney A, et al. . Risk of bias in non-randomized studies - of exposure (ROBINS-E). 2022. Available: https://www.riskofbias.info/welcome/robins-e-tool
    1. Katkade VB, Sanders KN, Zou KH. Real world data: an opportunity to supplement existing evidence for the use of long-established medicines in health care decision making. Multidiscip Healthc. 2018. Available: https://www.tandfonline.com/doi/full/10.2147/JMDH.S160029 - PMC - PubMed
    1. Baumfeld Andre E, Carrington N, Siami FS, et al. . The Current Landscape and Emerging Applications for Real-World Data in Diagnostics and Clinical Decision Support and its Impact on Regulatory Decision Making. Clin Pharmacol Ther 2022;112:1172–82. 10.1002/cpt.2565 Available: https://ascpt.onlinelibrary.wiley.com/doi/full/10.1002/cpt.2565 - DOI - PMC - PubMed

Publication types

LinkOut - more resources