Randomized Controlled Trial

. 2016 Jan 5;188(1):25-32.

doi: 10.1503/cmaj.150430. Epub 2015 Oct 26.

Do clinicians understand the size of treatment effects? A randomized survey across 8 countries

Bradley C Johnston¹, Pablo Alonso-Coello², Jan O Friedrich², Reem A Mustafa², Kari A O Tikkinen², Ignacio Neumann², Per O Vandvik², Elie A Akl², Bruno R da Costa², Neill K Adhikari², Gemma Mas Dalmau², Elise Kosunen², Jukka Mustonen², Mark W Crawford², Lehana Thabane², Gordon H Guyatt²

Affiliations

¹ Department of Anesthesia and Pain Medicine (Johnston, Crawford), The Hospital for Sick Children, University of Toronto, Toronto, Ont.; Institute of Health Policy, Management and Evaluation (Johnston), Dalla Lana School of Public Health, University of Toronto, Toronto, Ont.; Child Health Evaluative Sciences (Johnston), The Hospital for Sick Children Research Institute, Toronto, Ont.; Iberoamerican Cochrane Center (Alonso-Coello, Dalmau), Biomedical Research Institute Sant Pau, CIBER Epidemiologiay Salud Publica, Barcelona, Spain; Departments of Critical Care and Medicine (Friedrich), St. Michael's Hospital, Toronto, Ont.; Department of Medicine and Interdepartmental Division of Critical Care (Friedrich), University of Toronto, Toronto, Ont.; Departments of Medicine and Biomedical and Health Informatics (Mustafa), University of Missouri-Kansas City, Kansas City, Mo.; Department of Clinical Epidemiology and Biostatistics (Mustafa, Tikkinen, Neumann, Akl, Thabane, Guyatt), McMaster University, Hamilton, Ont.; Departments of Urology and Public Health (Tikkinen), Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland; Department of Internal Medicine (Neumann), Pontificia Universidad Catolica de Chile, Santiago, Chile; Institute of Health and Society (Vandvik), Faculty of Medicine, University of Oslo, Oslo, Norway; Department of Medicine (Vandvik), Innlandet Hospital Trust, Division Gjövik, Norway; Clinical Epidemiology Unit (Akl), American University of Beirut, Beirut, Lebanon; Institute of Primary Health Care (da Costa), University of Bern, Bern, Switzerland; Department of Critical Care Medicine and Sunnybrook Research Institute (Adhikari), Sunnybrook Health Sciences Centre, Toronto, Ont.; Interdepartmental Division of Critical Care (Adhikari), University of Toronto, Toronto, Ont.; School of Medicine and Centre for General Practice (Kosunen), University of Tampere and Pirkanmaa Hospital District, Tampere, Finland; School of Medicine and Department of Internal Medicine (Mustonen), University of Tampere and Tampere University Hospital, Tampere, Finland; Physiology and Experimental Medicine (Crawford), The Hospital for Sick Children Research Institute, Toronto, Ont.; Biostatistics Unit of the Centre for Evaluation of Medicines (Thabane), McMaster University, Hamilton, Ont.; Population Health Research Unit, Hamilton Health Sciences (Thabane), McMaster University, Hamilton, Ont.; Department of Medicine (Guyatt), McMaster University, Hamilton, Ont. bradley.johnston@sickkids.ca.
² Department of Anesthesia and Pain Medicine (Johnston, Crawford), The Hospital for Sick Children, University of Toronto, Toronto, Ont.; Institute of Health Policy, Management and Evaluation (Johnston), Dalla Lana School of Public Health, University of Toronto, Toronto, Ont.; Child Health Evaluative Sciences (Johnston), The Hospital for Sick Children Research Institute, Toronto, Ont.; Iberoamerican Cochrane Center (Alonso-Coello, Dalmau), Biomedical Research Institute Sant Pau, CIBER Epidemiologiay Salud Publica, Barcelona, Spain; Departments of Critical Care and Medicine (Friedrich), St. Michael's Hospital, Toronto, Ont.; Department of Medicine and Interdepartmental Division of Critical Care (Friedrich), University of Toronto, Toronto, Ont.; Departments of Medicine and Biomedical and Health Informatics (Mustafa), University of Missouri-Kansas City, Kansas City, Mo.; Department of Clinical Epidemiology and Biostatistics (Mustafa, Tikkinen, Neumann, Akl, Thabane, Guyatt), McMaster University, Hamilton, Ont.; Departments of Urology and Public Health (Tikkinen), Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland; Department of Internal Medicine (Neumann), Pontificia Universidad Catolica de Chile, Santiago, Chile; Institute of Health and Society (Vandvik), Faculty of Medicine, University of Oslo, Oslo, Norway; Department of Medicine (Vandvik), Innlandet Hospital Trust, Division Gjövik, Norway; Clinical Epidemiology Unit (Akl), American University of Beirut, Beirut, Lebanon; Institute of Primary Health Care (da Costa), University of Bern, Bern, Switzerland; Department of Critical Care Medicine and Sunnybrook Research Institute (Adhikari), Sunnybrook Health Sciences Centre, Toronto, Ont.; Interdepartmental Division of Critical Care (Adhikari), University of Toronto, Toronto, Ont.; School of Medicine and Centre for General Practice (Kosunen), University of Tampere and Pirkanmaa Hospital District, Tampere, Finland; School of Medicine and Department of Internal Medicine (Mustonen), University of Tampere and Tampere University Hospital, Tampere, Finland; Physiology and Experimental Medicine (Crawford), The Hospital for Sick Children Research Institute, Toronto, Ont.; Biostatistics Unit of the Centre for Evaluation of Medicines (Thabane), McMaster University, Hamilton, Ont.; Population Health Research Unit, Hamilton Health Sciences (Thabane), McMaster University, Hamilton, Ont.; Department of Medicine (Guyatt), McMaster University, Hamilton, Ont.

PMID: 26504102
PMCID: PMC4695351
DOI: 10.1503/cmaj.150430

Randomized Controlled Trial

Do clinicians understand the size of treatment effects? A randomized survey across 8 countries

Bradley C Johnston et al. CMAJ. 2016.

. 2016 Jan 5;188(1):25-32.

doi: 10.1503/cmaj.150430. Epub 2015 Oct 26.

Authors

Affiliations

¹ Department of Anesthesia and Pain Medicine (Johnston, Crawford), The Hospital for Sick Children, University of Toronto, Toronto, Ont.; Institute of Health Policy, Management and Evaluation (Johnston), Dalla Lana School of Public Health, University of Toronto, Toronto, Ont.; Child Health Evaluative Sciences (Johnston), The Hospital for Sick Children Research Institute, Toronto, Ont.; Iberoamerican Cochrane Center (Alonso-Coello, Dalmau), Biomedical Research Institute Sant Pau, CIBER Epidemiologiay Salud Publica, Barcelona, Spain; Departments of Critical Care and Medicine (Friedrich), St. Michael's Hospital, Toronto, Ont.; Department of Medicine and Interdepartmental Division of Critical Care (Friedrich), University of Toronto, Toronto, Ont.; Departments of Medicine and Biomedical and Health Informatics (Mustafa), University of Missouri-Kansas City, Kansas City, Mo.; Department of Clinical Epidemiology and Biostatistics (Mustafa, Tikkinen, Neumann, Akl, Thabane, Guyatt), McMaster University, Hamilton, Ont.; Departments of Urology and Public Health (Tikkinen), Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland; Department of Internal Medicine (Neumann), Pontificia Universidad Catolica de Chile, Santiago, Chile; Institute of Health and Society (Vandvik), Faculty of Medicine, University of Oslo, Oslo, Norway; Department of Medicine (Vandvik), Innlandet Hospital Trust, Division Gjövik, Norway; Clinical Epidemiology Unit (Akl), American University of Beirut, Beirut, Lebanon; Institute of Primary Health Care (da Costa), University of Bern, Bern, Switzerland; Department of Critical Care Medicine and Sunnybrook Research Institute (Adhikari), Sunnybrook Health Sciences Centre, Toronto, Ont.; Interdepartmental Division of Critical Care (Adhikari), University of Toronto, Toronto, Ont.; School of Medicine and Centre for General Practice (Kosunen), University of Tampere and Pirkanmaa Hospital District, Tampere, Finland; School of Medicine and Department of Internal Medicine (Mustonen), University of Tampere and Tampere University Hospital, Tampere, Finland; Physiology and Experimental Medicine (Crawford), The Hospital for Sick Children Research Institute, Toronto, Ont.; Biostatistics Unit of the Centre for Evaluation of Medicines (Thabane), McMaster University, Hamilton, Ont.; Population Health Research Unit, Hamilton Health Sciences (Thabane), McMaster University, Hamilton, Ont.; Department of Medicine (Guyatt), McMaster University, Hamilton, Ont. bradley.johnston@sickkids.ca.
² Department of Anesthesia and Pain Medicine (Johnston, Crawford), The Hospital for Sick Children, University of Toronto, Toronto, Ont.; Institute of Health Policy, Management and Evaluation (Johnston), Dalla Lana School of Public Health, University of Toronto, Toronto, Ont.; Child Health Evaluative Sciences (Johnston), The Hospital for Sick Children Research Institute, Toronto, Ont.; Iberoamerican Cochrane Center (Alonso-Coello, Dalmau), Biomedical Research Institute Sant Pau, CIBER Epidemiologiay Salud Publica, Barcelona, Spain; Departments of Critical Care and Medicine (Friedrich), St. Michael's Hospital, Toronto, Ont.; Department of Medicine and Interdepartmental Division of Critical Care (Friedrich), University of Toronto, Toronto, Ont.; Departments of Medicine and Biomedical and Health Informatics (Mustafa), University of Missouri-Kansas City, Kansas City, Mo.; Department of Clinical Epidemiology and Biostatistics (Mustafa, Tikkinen, Neumann, Akl, Thabane, Guyatt), McMaster University, Hamilton, Ont.; Departments of Urology and Public Health (Tikkinen), Helsinki University Central Hospital and University of Helsinki, Helsinki, Finland; Department of Internal Medicine (Neumann), Pontificia Universidad Catolica de Chile, Santiago, Chile; Institute of Health and Society (Vandvik), Faculty of Medicine, University of Oslo, Oslo, Norway; Department of Medicine (Vandvik), Innlandet Hospital Trust, Division Gjövik, Norway; Clinical Epidemiology Unit (Akl), American University of Beirut, Beirut, Lebanon; Institute of Primary Health Care (da Costa), University of Bern, Bern, Switzerland; Department of Critical Care Medicine and Sunnybrook Research Institute (Adhikari), Sunnybrook Health Sciences Centre, Toronto, Ont.; Interdepartmental Division of Critical Care (Adhikari), University of Toronto, Toronto, Ont.; School of Medicine and Centre for General Practice (Kosunen), University of Tampere and Pirkanmaa Hospital District, Tampere, Finland; School of Medicine and Department of Internal Medicine (Mustonen), University of Tampere and Tampere University Hospital, Tampere, Finland; Physiology and Experimental Medicine (Crawford), The Hospital for Sick Children Research Institute, Toronto, Ont.; Biostatistics Unit of the Centre for Evaluation of Medicines (Thabane), McMaster University, Hamilton, Ont.; Population Health Research Unit, Hamilton Health Sciences (Thabane), McMaster University, Hamilton, Ont.; Department of Medicine (Guyatt), McMaster University, Hamilton, Ont.

PMID: 26504102
PMCID: PMC4695351
DOI: 10.1503/cmaj.150430

Abstract

Background: Meta-analyses of continuous outcomes typically provide enough information for decision-makers to evaluate the extent to which chance can explain apparent differences between interventions. The interpretation of the magnitude of these differences - from trivial to large - can, however, be challenging. We investigated clinicians' understanding and perceptions of usefulness of 6 statistical formats for presenting continuous outcomes from meta-analyses (standardized mean difference, minimal important difference units, mean difference in natural units, ratio of means, relative risk and risk difference).

Methods: We invited 610 staff and trainees in internal medicine and family medicine programs in 8 countries to participate. Paper-based, self-administered questionnaires presented summary estimates of hypothetical interventions versus placebo for chronic pain. The estimates showed either a small or a large effect for each of the 6 statistical formats for presenting continuous outcomes. Questions addressed participants' understanding of the magnitude of treatment effects and their perception of the usefulness of the presentation format. We randomly assigned participants 1 of 4 versions of the questionnaire, each with a different effect size (large or small) and presentation order for the 6 formats (1 to 6, or 6 to 1).

Results: Overall, 531 (87.0%) of the clinicians responded. Respondents best understood risk difference, followed by relative risk and ratio of means. Similarly, they perceived the dichotomous presentation of continuous outcomes (relative risk and risk difference) to be most useful. Presenting results as a standardized mean difference, the longest standing and most widely used approach, was poorly understood and perceived as least useful.

Interpretation: None of the presentation formats were well understood or perceived as extremely useful. Clinicians best understood the dichotomous presentations of continuous outcomes and perceived them to be the most useful. Further initiatives to help clinicians better grasp the magnitude of the treatment effect are needed.

PubMed Disclaimer

Figures

**Figure 1:**
Respondents’ understanding of the magnitude of the treatment effect for each of 6 statistical formats used to present continuous outcomes from meta-analyses. Higher percentages represent greater understanding; error bars = 95% confidence intervals. Mean difference = mean difference in natural units, MID = minimal important difference, SMD = standardized mean difference.

**Figure 2:**
Perceived usefulness of each statistical format for clinical decision-making. Higher scores represent higher perceived usefulness; error bars = 95% confidence intervals. Mean difference = mean difference in natural units, MID = minimal important difference, SMD = standardized mean difference.

**Figure 3:**
Factors associated with respondents’ understanding of the statistical formats. An odds ratio below 1.0 indicates a worse understanding than the reference category. CI = confidence interval, mean difference = mean difference in natural units, MID = minimal important difference, SMD = standardized mean difference.

See this image and copyright information in PMC

Comment in

Physicians and knowledge translation of statistics: Mind the gap.
Bartlett G, Gagnon J. Bartlett G, et al. CMAJ. 2016 Jan 5;188(1):11-12. doi: 10.1503/cmaj.151254. Epub 2015 Nov 30. CMAJ. 2016. PMID: 26621998 Free PMC article. No abstract available.
Minimal important difference is important.
Shrier I. Shrier I. CMAJ. 2016 Apr 5;188(6):451. doi: 10.1503/cmaj.1150088. CMAJ. 2016. PMID: 27044791 Free PMC article. No abstract available.

Cited by

Minimal important difference is important.
Shrier I. Shrier I. CMAJ. 2016 Apr 5;188(6):451. doi: 10.1503/cmaj.1150088. CMAJ. 2016. PMID: 27044791 Free PMC article. No abstract available.
Efficacy and safety of low and very low carbohydrate diets for type 2 diabetes remission: systematic review and meta-analysis of published and unpublished randomized trial data.
Goldenberg JZ, Day A, Brinkworth GD, Sato J, Yamada S, Jönsson T, Beardsley J, Johnson JA, Thabane L, Johnston BC. Goldenberg JZ, et al. BMJ. 2021 Jan 13;372:m4743. doi: 10.1136/bmj.m4743. BMJ. 2021. PMID: 33441384 Free PMC article.
Foundational Statistical Principles in Medical Research: A Tutorial on Odds Ratios, Relative Risk, Absolute Risk, and Number Needed to Treat.
Monaghan TF, Rahman SN, Agudelo CW, Wein AJ, Lazar JM, Everaert K, Dmochowski RR. Monaghan TF, et al. Int J Environ Res Public Health. 2021 May 25;18(11):5669. doi: 10.3390/ijerph18115669. Int J Environ Res Public Health. 2021. PMID: 34070675 Free PMC article. Review.
Visual data: a new tool to improve the presentation of clinical trial results.
Ritti-Dias RM, Correia MA, Farah BQ. Ritti-Dias RM, et al. Einstein (Sao Paulo). 2019 Nov 25;18:eAE4729. doi: 10.31744/einstein_journal/2020AE4729. eCollection 2020. Einstein (Sao Paulo). 2019. PMID: 31778465 Free PMC article.
Comprendre et expliquer le risque: Mesurer les résultats et l’ampleur des bienfaits et des préjudices.
Bell NR, Dickinson JA, Grad R, Singh H, Kasperavicius D, Thombs BD. Bell NR, et al. Can Fam Physician. 2018 Mar;64(3):186-191. Can Fam Physician. 2018. PMID: 29540383 Free PMC article. French. No abstract available.

See all "Cited by" articles

References

1. Guyatt GH, Osoba D, Wu AW, et al. Methods to explain the clinical significance of health status measures. Mayo Clin Proc 2002;77:371–83. - PubMed
1. Fransen M, McConnell S. Exercise for osteoarthritis of the knee. Cochrane Database Syst Rev 2008;(4):CD004376. - PubMed
1. Juhl C, Lund H, Roos EM, et al. A hierarchy of patient-reported outcomes for meta-analysis of knee osteoarthritis trials: empirical evidence from a survey of high impact journals. Arthritis 2012;2012:136245. - PMC - PubMed
1. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. London (UK): Cochrane Collaboration; 2011. Available: www.cochrane-handbook.org (accessed 2015 May 29).
1. Van Den Noortgate W, Onghena P. Estimating the mean effect size in meta-analysis: bias, precision, and mean squared error of different weighting methods. Behav Res Methods Instrum Comput 2003;35:504–11. - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Do clinicians understand the size of treatment effects? A randomized survey across 8 countries

Affiliations

Do clinicians understand the size of treatment effects? A randomized survey across 8 countries

Authors

Affiliations

Abstract

Figures

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Other Literature Sources