Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Feb 21;27(1):e300978.
doi: 10.1136/bmjment-2023-300978.

Understanding effect size: an international online survey among psychiatrists, psychologists, physicians from other medical specialities, dentists and other health professionals

Ferdinand Heimke  1 , Yuki Furukawa  2 , Spyridon Siafis  1   3 , Bradley C Johnston  4   5 , Rolf R Engel  6 , Toshi A Furukawa  7 , Stefan Leucht  8   3
Affiliations

Understanding effect size: an international online survey among psychiatrists, psychologists, physicians from other medical specialities, dentists and other health professionals

Ferdinand Heimke et al. BMJ Ment Health. .

Abstract

Background and objective: Various ways exist to display the effectiveness of medical treatment options. This study examined various psychiatric, medical and allied professionals' understanding and perceived usefulness of eight effect size indices for presenting both dichotomous and continuous outcome data.

Methods: We surveyed 1316 participants from 13 countries using an online questionnaire. We presented hypothetical treatment effects of interventions versus placebo concerning chronic pain using eight different effect size measures. For each index, the participants had to judge the magnitude of the shown effect, to indicate how certain they felt about their own answer and how useful they found the given effect size index.

Findings: Overall, 762 (57.9%) participants fully completed the questionnaire. In terms of understanding, the best results emerged when both the control event rate (CER) and the experimental event rate (EER) were presented. The difference in minimal importance difference units (MID unit) was understood worst. Respondents also found CER and EER to be the most useful presentation approach while they rated MID unit as the least useful. Confidence in the risk ratio ranked high, even though it was rather poorly understood.

Conclusions and clinical implications: For dichotomous outcomes, presenting the effects in terms of the CER and EER could lead to the most correct interpretation. Relative measures including the risk ratio must be supplemented with absolute measures such as the CER and EER. Effects on continuous outcomes were better understood through standardised mean differences than mean differences. These can also be supplemented by dichotomised CER and EER.

Keywords: Data Interpretation, Statistical.

PubMed Disclaimer

Conflict of interest statement

Competing interests: In the last 3 years, SL has received honoraria as a consultant and/or advisor and/or for lectures and/or for educational material from Alkermes, Angelini, Eisai, Gedeon Richter, Janssen, Lundbeck, Medichem, Medscape, Merck Sharpp and Dome, Mitshubishi, Neurotorium, NovoNordisk, Otsuka, Recordati, Roche, Rovi, Sanofi Aventis, TEVA. TAF reports personal fees from Boehringer-Ingelheim, DT Axis, Kyoto University Original, Shionogi and SONY and UpToDate, and a grant from Shionogi, outside the submitted work; In addition, TAF has patents 2020-548587 and 2022-082495 pending, and intellectual properties for Kokoro-app licensed to Mitsubishi-Tanabe.

Figures

Figure 1
Figure 1
Proportion of correct answers, perceived confidence and perceived usefulness description. (A) Proportion of correct answers regarding the estimation of the size of treatment effects presented by the eight effect size measures. (B) Participants’ perceived confidence (on a scale between 1 and 7) while dealing with the effect size measures. A higher value stands for higher confidence. (C) Participants’ perceived usefulness (on a scale between 1 and 7) for the effect size measures. A higher value stands for higher perceived usefulness. Error bars=95% CI. CER & EER, control event rate and experimental event rate; MD, mean difference; MID unit, difference in minimal importance difference units; NNT, number needed to treat; RD, risk difference; RoM, ratio of means; RR, risk ratio; SMD, standardised mean difference.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Leucht S, Siafis S, Engel RR, et al. . How efficacious are antipsychotic drugs for schizophrenia? An interpretation based on 13 effect size indices. Schizophr Bull 2022;48:27–36. 10.1093/schbul/sbab094 - DOI - PMC - PubMed
    1. Guyatt GH, Thorlund K, Oxman AD, et al. . GRADE guidelines: 13. preparing summary of findings tables and evidence profiles—continuous outcomes. J Clin Epidemiol 2013;66:173–83. 10.1016/j.jclinepi.2012.08.001 - DOI - PubMed
    1. Valentine JC, Aloe AM. How to communicate effect sizes for continuous outcomes: a review of existing options and introducing a new metric. J Clin Epidemiol 2016;72:84–9. 10.1016/j.jclinepi.2015.10.017 - DOI - PubMed
    1. Higgins JPT, Li T, Deeks JJ. Chapter 6: choosing effect measures and computing estimates of effect. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022). Cochrane, 2022. Available: www.training.cochrane.org/handbook
    1. Friedrich JO, Adhikari NKJ, Beyene J. Ratio of means for analyzing continuous outcomes in meta-analysis performed as well as mean difference methods. J Clin Epidemiol 2011;64:556–64. 10.1016/j.jclinepi.2010.09.016 - DOI - PubMed