Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Oct 16;22(1):708.
doi: 10.1186/s13063-021-05686-8.

Confidence interval of risk difference by different statistical methods and its impact on the study conclusion in antibiotic non-inferiority trials

Anthony D Bai  1   2 , Adam S Komorowski  3   4 , Carson K L Lo  5 , Pranav Tandon  6 , Xena X Li  4   5 , Vaibhav Mokashi  5 , Anna Cvetkovic  5 , Aidan Findlater  5 , Laurel Liang  7 , George Tomlinson  8   9 , Mark Loeb  5 , Dominik Mertz  5 ; McMaster Infectious Diseases Fellow Research Group
Affiliations

Confidence interval of risk difference by different statistical methods and its impact on the study conclusion in antibiotic non-inferiority trials

Anthony D Bai et al. Trials. .

Abstract

Background: Numerous statistical methods can be used to calculate the confidence interval (CI) of risk differences. There is consensus in previous literature that the Wald method should be discouraged. We compared five statistical methods for estimating the CI of risk difference in terms of CI width and study conclusion in antibiotic non-inferiority trials.

Methods: In a secondary analysis of a systematic review, we included non-inferiority trials that compared different antibiotic regimens, reported risk differences for the primary outcome, and described the number of successes and/or failures as well as patients in each arm. For each study, we re-calculated the risk difference CI using the Wald, Agresti-Caffo, Newcombe, Miettinen-Nurminen, and skewness-corrected asymptotic score (SCAS) methods. The CIs by different statistical methods were compared in terms of CI width and conclusion on non-inferiority. A wider CI was considered to be more conservative.

Results: The analysis included 224 comparisons from 213 studies. The statistical method used to calculate CI was not reported in 134 (59.8%) cases. The median (interquartile range IQR) for CI width by Wald, Agresti-Caffo, Newcombe, Miettinen-Nurminen, and SCAS methods was 13.0% (10.8%, 17.4%), 13.3% (10.9%, 18.5%), 13.6% (11.1%, 18.9%), 13.6% (11.1% and 19.0%), and 13.4% (11.1%, 18.9%), respectively. In 216 comparisons that reported a non-inferiority margin, the conclusion on non-inferiority was the same across the five statistical methods in 211 (97.7%) cases. The differences in CI width were more in trials with a sample size of 100 or less in each group and treatment success rate above 90%. Of the 18 trials in this subgroup with a specified non-inferiority margin, non-inferiority was shown in 17 (94.4%), 16 (88.9%), 14 (77.8%), 14 (77.8%), and 15 (83.3%) cases based on CI by Wald, Agresti-Caffo, Newcombe, Miettinen-Nurminen, and SCAS methods, respectively.

Conclusions: The statistical method used to calculate CI was not reported in the majority of antibiotic non-inferiority trials. Different statistical methods for CI resulted in different conclusions on non-inferiority in 2.3% cases. The differences in CI widths were highest in trials with a sample size of 100 or less in each group and a treatment success rate above 90%.

Trial registration: PROSPERO CRD42020165040 . April 28, 2020.

Keywords: Confidence interval; Non-inferiority trials; Risk differences; Statistics.

PubMed Disclaimer

Conflict of interest statement

The authors declare that there is no conflict of interest.

Figures

Fig. 1
Fig. 1
Relationship of confidence interval width to sample size. A-C, Agresti-Caffo method; CI, confidence interval; M-N, Miettinen-Nurminen method; SCAS, skewness-corrected asymptotic score method. Each line is a smooth fitted line of data points for each statistical method. After sample size in each group increases to more than 100, the 5 methods converge to even smaller differences that are not shown in this figure
Fig. 2
Fig. 2
Differences in confidence interval compared to Wald method in relation to sample size and stratified by success rate. A-C, Agresti-Caffo method; CI, confidence interval; M-N, Miettinen-Nurminen method; SCAS, skewness-corrected asymptotic score method. The CI width difference is the method of interest minus the Wald CI. Each line is a smooth fitted line of data points for each statistical method. Note that the y-axis scale is different
Fig. 3
Fig. 3
Agreement across confidence interval methods for different non-inferiority margins. CI, confidence interval
Fig. 4
Fig. 4
Percentage of non-inferiority shown by different confidence interval methods for different non-inferiority margins. A-C, Agresti-Caffo method; CI, confidence interval; M-N, Miettinen-Nurminen method; NIM, non-inferiority margin; SCAS, skewness-corrected asymptotic score method
See this image and copyright information in PMC

References

    1. Piaggio G, Elbourne DR, Pocock SJ, Evans SJW, Altman DG, CONSORT Group Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. JAMA. 2012;308(24):2594–2604. doi: 10.1001/jama.2012.87802. - DOI - PubMed
    1. Altman DG, Machin D, Bryant TN, et al., editors. Statistics with confidence. 2. London: BMJ Books; 2000. pp. 48–49.
    1. Fagerland MW, Lydersen S, Laake P. Recommended confidence intervals for two independent binomial proportions. Stat Methods Med Res. 2015;24(2):224–254. doi: 10.1177/0962280211415469. - DOI - PubMed
    1. Agresti A, Caffo B. Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. Am Stat. 2000;54:280–288.
    1. Newcombe RG. Interval estimation for the difference between independent proportions: comparison of eleven methods. Stat Med. 1998;17(8):873–890. doi: 10.1002/(SICI)1097-0258(19980430)17:8<873::AID-SIM779>3.0.CO;2-I. - DOI - PubMed

Publication types

Substances