The methodological quality of systematic reviews regarding the Core Outcome Set (COS) development

Abstract

Background: The Core Outcome Measures in Effectiveness Trials (COMET) working group proposed core outcome sets (COS) to address the heterogeneity in outcome measures in clinical studies. According to the recommendations of COMET, performing systematic reviews (SRs) usually was the first step for COS development. However, the SRs that serve as a basis for COS are not specifically appraised by organizations such as COMET regarding their quality. Here, we investigated the status of SRs related to development of COS and evaluated their methodological quality.

Methods: We conducted a search on PubMed to identify SRs related to COS development published from inception to May 2022. We qualitatively summarized the disease included in SR topics, and the studies included in the SRs. We evaluated the methodological quality of the SRs using AMSTAR 2.0 and compared the overall quality of SRs with and without protocols using the Mann-Whitney U test.

Results: We included 175 SRs from 23 different countries or regions, and they mainly focused on five diseases: musculoskeletal system or connective tissue disease (n = 19, 10.86%), injury, poisoning, or certain other consequences of external causes (n = 18, 10.29%), digestive system disease (n = 16, 9.14%), nervous system disease (n = 15, 8.57%), and genitourinary system disease (n = 15, 8.57%). Although 88.00% of SRs included randomized controlled trials (RCTs), only a few SRs (23.38%) employed appropriate tools to assess the risk of bias in RCTs. The assessment results on the basis of AMSTAR 2.0 indicated that most SRs (93.71%) were rated as ''critically low'' to ''low'' in terms of overall confidence. The overall confidence of SRs with protocols was significantly higher than that without protocols (P <.001). Compared to the SRs with protocols on Core Outcome Measures in Effectiveness Trials (COMET), SRs with protocols on PROSPERO were of better overall confidence (P = .017).

Conclusion: The overall quality of published SRs regarding COS development was poor. Our findings emphasize the need for researchers to carefully select the disease topic and strictly adhere to the requirements of optimal methodology when conducting a SR for the establishment of a COS.

Keywords: AMSATR 2.0; Core outcome measures in effectiveness trials (COMET); Core outcome set (COS); Methodological research; Systematic reviews (SRs).

Fig. 1

Flow diagram of study selection process for this study. SR: systematic review; COS: core outcome set

Fig. 2

Disease distribution for constructing core outcome set

Fig. 3

Design types of studies included in the systematic reviews. QS: Qualitative methods; SR: systematic review; RCT: randomized controlled trial; Quasi-RCT: quasi-randomized controlled trial; Non-RCT: non-randomized controlled trial; OS: Observational study

Fig. 4

Methodological pathway for developing core outcome set. Pathway 1: “(1)systematic review, (2)semi-structured interview, (3)Delphi survey, (4)consensus meeting”; Pathway 2: “(1)systematic review, (2)Delphi survey”; Pathway 3: “(1)systematic review, (2)Delphi survey, (3)consensus meeting”; Pathway 4: “(1)systematic review, (2) focus group, (3)Delphi survey, (4)consensus meeting”; Pathway 5: “(1)systematic review, (2)semi-structured interview, (3)focus group, (4)Delphi survey, (5)consensus meeting”; Pathway 6: “(1)systematic review, (2)semi-structured interview, (3)focus group, (4)Delphi survey/consensus meeting”; Pathway 7: “(1)systematic review, (2)focus group, (3) consensus meeting”

Fig. 5

Methodological quality of included systematic reviews according to the AMSTAR 2.0. Green: “yes”; Blue: “partially yes”; Red: “no”; Grey: “not applicable due to no meta-analysis conducted”