Systematic reviews experience major limitations in reporting absolute effects

Pablo Alonso-Coello¹, Alonso Carrasco-Labra², Romina Brignardello-Petersen³, Ignacio Neumann⁴, Elie A Akl⁵, Robin W M Vernooij⁶, Brad C Johnston⁷, Xin Sun⁸, Matthias Briel⁹, Jason W Busse¹⁰, Shanil Ebrahim¹¹, Carlos E Granados¹², Alfonso Iorio¹³, Affan Irfan¹⁴, Laura Martínez García⁶, Reem A Mustafa¹⁵, Anggie Ramírez-Morera¹⁶, Anna Selva⁶, Ivan Solà⁶, Andrea Juliana Sanabria⁶, Kari A O Tikkinen¹⁷, Per Olav Vandvik¹⁸, Oscar E Zazueta⁶, Yuqing Zhang¹⁹, Qi Zhou²⁰, Holger Schünemann²¹, Gordon H Guyatt²¹

Affiliations

¹ Clinical Epidemiology and Public Health Department, Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau (IIB Sant Pau), C/ Sant Antoni Maria Claret, 167, Pavelló 18, planta 0, 08025 Barcelona, Spain; Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública (CIBERESP); Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada. Electronic address: PAlonso@santpau.cat.
² Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Evidence-Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Sergio Livingstone 943, Santiago, Chile.
³ Evidence-Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Sergio Livingstone 943, Santiago, Chile; Institute of Health Policy, Management & Evaluation, University of Toronto, 4th Floor, 155 College St, Toronto, Ontario M5T 3M6, Canada.
⁴ Department of Medicine, Pontificia Universidad Católica de Chile, Avenida Libertador Bernardo O Higgins 340, Santiago, Región Metropolitana, Chile.
⁵ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Internal Medicine, American University of Beirut Medical Center, P.O. Box: 11-0236, Riad-El-Solh, Beirut 1107 2020.
⁶ Clinical Epidemiology and Public Health Department, Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau (IIB Sant Pau), C/ Sant Antoni Maria Claret, 167, Pavelló 18, planta 0, 08025 Barcelona, Spain.
⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Institute of Health Policy, Management & Evaluation, University of Toronto, 4th Floor, 155 College St, Toronto, Ontario M5T 3M6, Canada; Department of Anesthesia & Pain Medicine, The Hospital for Sick Children, 555 University Ave., Toronto, Ontario M5G 1X8, Canada.
⁸ Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, 37 Guo Xue Xiang, Chengdu 610041, China.
⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, Spitalstrasse 12, 4031 Basel, Switzerland.
¹⁰ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Anesthesia, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Michael G. DeGroote Institute for Pain Research and Care, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
¹¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, 1265 Welch Road, Stanford, CA 94305, USA.
¹² Área de investigaciones, Facultad de Medicina, Universidad de La Sabana, Campus del Puente del Común Km, 7 Autopista Norte, Chía, Colombia.
¹³ Department of Medicine, McMaster University, 1280 Main St W, Hamilton, Ontario L8S 4L8, Canada.
¹⁴ Department of Cardiology, University of Louisville, Rudd Heart & Lung Building, 201 Abraham Flexner Way, Suite 600, Louisville, KY 40292, USA.
¹⁵ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Departments of Medicine and Biomedical & Health Informatics, University of Missouri-Kansas City, School of Medicine, M4-303, 2411 Holmes St, Kansas City, MO, USA.
¹⁶ CCSS Permanent Medical Advisor, Health Care Development Division, IHCAI Foundation & Central America Cochrane, 1st Ave., 35th and 37th St, Number 3530, Barrio Escalante, San José, Costa Rica.
¹⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Departments of Urology and Public Health, Helsinki University Hospital and University of Helsinki, Haartmaninkatu 4, 00029 Helsinki, Finland.
¹⁸ Institute of Health and Society, Faculty of Medicine, University of Oslo, P.O. Box 1130 Blindern, 0318 Oslo, Norway; Guang'anmen Hospital China Academy of Chinese Medical Science, No. 5 Bei Xian Ge St, Xuan Wu District, Beijing 100053, China.
¹⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Guang'anmen Hospital China Academy of Chinese Medical Science, No. 5 Bei Xian Ge St, Xuan Wu District, Beijing 100053, China.
²⁰ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
²¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, McMaster University, 1280 Main St W, Hamilton, Ontario L8S 4L8, Canada.

PMID: 26560992
DOI: 10.1016/j.jclinepi.2015.11.002

Meta-Analysis

Systematic reviews experience major limitations in reporting absolute effects

Pablo Alonso-Coello et al. J Clin Epidemiol. 2016 Apr.

. 2016 Apr:72:16-26.

doi: 10.1016/j.jclinepi.2015.11.002. Epub 2015 Nov 10.

Authors

Affiliations

¹ Clinical Epidemiology and Public Health Department, Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau (IIB Sant Pau), C/ Sant Antoni Maria Claret, 167, Pavelló 18, planta 0, 08025 Barcelona, Spain; Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública (CIBERESP); Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada. Electronic address: PAlonso@santpau.cat.
² Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Evidence-Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Sergio Livingstone 943, Santiago, Chile.
³ Evidence-Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Sergio Livingstone 943, Santiago, Chile; Institute of Health Policy, Management & Evaluation, University of Toronto, 4th Floor, 155 College St, Toronto, Ontario M5T 3M6, Canada.
⁴ Department of Medicine, Pontificia Universidad Católica de Chile, Avenida Libertador Bernardo O Higgins 340, Santiago, Región Metropolitana, Chile.
⁵ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Internal Medicine, American University of Beirut Medical Center, P.O. Box: 11-0236, Riad-El-Solh, Beirut 1107 2020.
⁶ Clinical Epidemiology and Public Health Department, Iberoamerican Cochrane Centre, Biomedical Research Institute Sant Pau (IIB Sant Pau), C/ Sant Antoni Maria Claret, 167, Pavelló 18, planta 0, 08025 Barcelona, Spain.
⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Institute of Health Policy, Management & Evaluation, University of Toronto, 4th Floor, 155 College St, Toronto, Ontario M5T 3M6, Canada; Department of Anesthesia & Pain Medicine, The Hospital for Sick Children, 555 University Ave., Toronto, Ontario M5G 1X8, Canada.
⁸ Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, 37 Guo Xue Xiang, Chengdu 610041, China.
⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, Spitalstrasse 12, 4031 Basel, Switzerland.
¹⁰ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Anesthesia, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Michael G. DeGroote Institute for Pain Research and Care, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
¹¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, 1265 Welch Road, Stanford, CA 94305, USA.
¹² Área de investigaciones, Facultad de Medicina, Universidad de La Sabana, Campus del Puente del Común Km, 7 Autopista Norte, Chía, Colombia.
¹³ Department of Medicine, McMaster University, 1280 Main St W, Hamilton, Ontario L8S 4L8, Canada.
¹⁴ Department of Cardiology, University of Louisville, Rudd Heart & Lung Building, 201 Abraham Flexner Way, Suite 600, Louisville, KY 40292, USA.
¹⁵ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Departments of Medicine and Biomedical & Health Informatics, University of Missouri-Kansas City, School of Medicine, M4-303, 2411 Holmes St, Kansas City, MO, USA.
¹⁶ CCSS Permanent Medical Advisor, Health Care Development Division, IHCAI Foundation & Central America Cochrane, 1st Ave., 35th and 37th St, Number 3530, Barrio Escalante, San José, Costa Rica.
¹⁷ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Departments of Urology and Public Health, Helsinki University Hospital and University of Helsinki, Haartmaninkatu 4, 00029 Helsinki, Finland.
¹⁸ Institute of Health and Society, Faculty of Medicine, University of Oslo, P.O. Box 1130 Blindern, 0318 Oslo, Norway; Guang'anmen Hospital China Academy of Chinese Medical Science, No. 5 Bei Xian Ge St, Xuan Wu District, Beijing 100053, China.
¹⁹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Guang'anmen Hospital China Academy of Chinese Medical Science, No. 5 Bei Xian Ge St, Xuan Wu District, Beijing 100053, China.
²⁰ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
²¹ Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, McMaster University, 1280 Main St W, Hamilton, Ontario L8S 4L8, Canada.

PMID: 26560992
DOI: 10.1016/j.jclinepi.2015.11.002

Abstract

Objectives: Expressing treatment effects in relative terms yields larger numbers than expressions in absolute terms, affecting the judgment of the clinicians and patients regarding the treatment options. It is uncertain how authors of systematic reviews (SRs) absolute effect estimates are reported in. We therefore undertook a systematic survey to identify and describe the reporting and methods for calculating absolute effect estimates in SRs.

Study design and setting: Two reviewers independently screened title, abstract, and full text and extracted data from a sample of Cochrane and non-Cochrane SRs. We used regression analyses to examine the association between study characteristics and the reporting of absolute estimates for the most patient-important outcome.

Results: We included 202 SRs (98 Cochrane and 104 non-Cochrane), most of which (92.1%) included standard meta-analyses including relative estimates of effect. Of the 202 SRs, 73 (36.1%) reported absolute effect estimates for the most patient-important outcome. SRs with statistically significant effects were more likely to report absolute estimates (odds ratio, 2.26; 95% confidence interval: 1.08, 4.74). The most commonly reported absolute estimates were: for each intervention, risk of adverse outcomes expressed as a percentage (41.1%); number needed to treat (26.0%); and risk for each intervention expressed as natural units or natural frequencies (24.7%). In 12.3% of the SRs that reported absolute effect estimates for both benefit and harm outcomes, harm outcomes were reported exclusively as absolute estimates. Exclusively reporting of beneficial outcomes as absolute estimates occurred in 6.8% of the SRs.

Conclusions: Most SRs do not report absolute effects. Those that do often report them inadequately, thus requiring users of SRs to generate their own estimates of absolute effects. For any apparently effective or harmful intervention, SR authors should report both absolute and relative estimates to optimize the interpretation of their findings.

Keywords: Absolute effect estimates; Absolute measures; Decision making; Framing; Reporting; Risk difference; Systematic review.

PubMed Disclaimer

Comment in

Which estimates comprise optimal reporting in Systematic Reviews?
Stovitz SD, Shrier I. Stovitz SD, et al. J Clin Epidemiol. 2017 Jan;81:140. doi: 10.1016/j.jclinepi.2016.05.013. Epub 2016 Jul 7. J Clin Epidemiol. 2017. PMID: 27394674 No abstract available.
RE: Which estimates comprise optimal reporting in systematic reviews?
Alonso-Coello P, Vernooij RW, Carrasco-Labra A, Vandvik P, Guyatt GH. Alonso-Coello P, et al. J Clin Epidemiol. 2017 Jan;81:140-141. doi: 10.1016/j.jclinepi.2017.01.001. J Clin Epidemiol. 2017. PMID: 28215276 No abstract available.

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- ClinicalKey
- Elsevier Science
Other Literature Sources
- scite Smart Citations
Research Materials
- NCI CPTC Antibody Characterization Program

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Systematic reviews experience major limitations in reporting absolute effects

Affiliations

Systematic reviews experience major limitations in reporting absolute effects

Authors

Affiliations

Abstract

Comment in

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Other Literature Sources

Research Materials