COVID-19-related medical research: a meta-research and critical appraisal

Abstract

Background: Since the start of the COVID-19 outbreak, a large number of COVID-19-related papers have been published. However, concerns about the risk of expedited science have been raised. We aimed at reviewing and categorizing COVID-19-related medical research and to critically appraise peer-reviewed original articles.

Methods: The data sources were Pubmed, Cochrane COVID-19 register study, arXiv, medRxiv and bioRxiv, from 01/11/2019 to 01/05/2020. Peer-reviewed and preprints publications related to COVID-19 were included, written in English or Chinese. No limitations were placed on study design. Reviewers screened and categorized studies according to i) publication type, ii) country of publication, and iii) topics covered. Original articles were critically appraised using validated quality assessment tools.

Results: Among the 11,452 publications identified, 10,516 met the inclusion criteria, among which 7468 (71.0%) were peer-reviewed articles. Among these, 4190 publications (56.1%) did not include any data or analytics (comprising expert opinion pieces). Overall, the most represented topics were infectious disease (n = 2326, 22.1%), epidemiology (n = 1802, 17.1%), and global health (n = 1602, 15.2%). The top five publishing countries were China (25.8%), United States (22.3%), United Kingdom (8.8%), Italy (8.1%) and India (3.4%). The dynamic of publication showed that the exponential growth of COVID-19 peer-reviewed articles was mainly driven by publications without original data (mean 261.5 articles ± 51.1 per week) as compared with original articles (mean of 69.3 ± 22.3 articles per week). Original articles including patient data accounted for 713 (9.5%) of peer-reviewed studies. A total of 576 original articles (80.8%) showed intermediate to high risk of bias. Last, except for simulation studies that mainly used large-scale open data, the median number of patients enrolled was of 102 (IQR = 37-337).

Conclusions: Since the beginning of the COVID-19 pandemic, the majority of research is composed by publications without original data. Peer-reviewed original articles with data showed a high risk of bias and included a limited number of patients. Together, these findings underscore the urgent need to strike a balance between the velocity and quality of research, and to cautiously consider medical information and clinical applicability in a pressing, pandemic context. SYSTEMATIC REVIEW REGISTRATION: https://osf.io/5zjyx/.

Keywords: COVID-19; Critical appraisal; Quality of research; Systematic review.

Fig. 1

Study flowchart. Flowchart based on preferred reporting for systematic reviews and meta-analyses (PRISMA) guidelines (supplementary method 1), depicting the review process and the inclusion/exclusion criteria. Pubmed and Cochrane COVID-19 register study were used for identifying peer-reviewed articles, and bioRxiv, medRxiv and arXiv were used for identifying preprints

Fig. 2

Dynamics of publication and trends in COVID-19 spread over time. This figure depicts the number of medical articles published and number of COVID-19 cases diagnosed worldwide over time (panel a) and the trends over time of peer-reviewed articles (N = 7468, panel b), categorized into publications without original data (N = 4190, 56.1%), original articles (N = 1109, 14.9%), cases reports (N = 697, 9.3%), reviews (N = 638, 8.6%), research letters (N = 786, 10.5%), and systematic reviews (N = 48, 0.6%)

Fig. 3

Number of COVID-19-related medical articles published by authors from 10 most productive countries. a All; b Peer-reviewed articles; c Preprints

Fig. 4

Topics addressed in COVID-19-related medical articles. Each barplot represents the number of articles dealing with the corresponding topic. 200 topics were listed after the first screening. After the discussion and consensus with our teams, they were categorized into 45 topics presented in the figure. All topics meaning is detailed in the supplementary Table 1. For a better insight, we have defined an ultimate categorization, defined in the legend: 1) Epidemiology, diagnosis, prognosis and treatment, 2) Health policy, 3) Modeling, 4) Medical specialty, and 5) Other