Evidence appraisal: a scoping review, conceptual framework, and research agenda

Abstract

Objective: Critical appraisal of clinical evidence promises to help prevent, detect, and address flaws related to study importance, ethics, validity, applicability, and reporting. These research issues are of growing concern. The purpose of this scoping review is to survey the current literature on evidence appraisal to develop a conceptual framework and an informatics research agenda.

Methods: We conducted an iterative literature search of Medline for discussion or research on the critical appraisal of clinical evidence. After title and abstract review, 121 articles were included in the analysis. We performed qualitative thematic analysis to describe the evidence appraisal architecture and its issues and opportunities. From this analysis, we derived a conceptual framework and an informatics research agenda.

Results: We identified 68 themes in 10 categories. This analysis revealed that the practice of evidence appraisal is quite common but is rarely subjected to documentation, organization, validation, integration, or uptake. This is related to underdeveloped tools, scant incentives, and insufficient acquisition of appraisal data and transformation of the data into usable knowledge.

Discussion: The gaps in acquiring appraisal data, transforming the data into actionable information and knowledge, and ensuring its dissemination and adoption can be addressed with proven informatics approaches.

Conclusions: Evidence appraisal faces several challenges, but implementing an informatics research agenda would likely help realize the potential of evidence appraisal for improving the rigor and value of clinical evidence.

Keywords: clinical research informatics; critical appraisal; journal clubs; journal comments; post-publication peer review.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram demonstrating article screening process. (A) During article screening and initial thematic analysis, additional search terms were identified and a further search was performed. (B) Screening was performed on the title and, as available, the abstract of retrieved articles, with full text reviewed if available and further ambiguity remained.

Figure 2.

Number of articles by publication year.

Figure 3.

Number of articles by publication type.

Figure 4.

Conceptual framework. This figure illustrates the architecture of evidence appraisal as described or discussed in the biomedical literature and with actual or potential informatics resources to enable key steps. Superscript includes relevant themes or theme categories. The line style indicates whether these elements were described to exist or not within the scoping review or to the authors’ knowledge.