Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Aug 3;3(8):e2012469.
doi: 10.1001/jamanetworkopen.2020.12469.

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results

Muhammad Shahzeb Khan  1 , Gregg C Fonarow  2 , Tim Friede  3 , Noman Lateef  4 , Safi U Khan  5 , Stefan D Anker  6 , Frank E Harrell Jr  7 , Javed Butler  8
Affiliations

Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results

Muhammad Shahzeb Khan et al. JAMA Netw Open. .

Abstract

Importance: Interpreting randomized clinical trials (RCTs) and their clinical relevance is challenging when P values are either marginally above or below the P = .05 threshold.

Objective: To use the concept of reverse fragility index (RFI) to provide a measure of confidence in the neutrality of RCT results when assessed from the clinical perspective.

Design, setting, and participants: In this cross-sectional study, a MEDLINE search was conducted for RCTs published from January 1, 2013, to December 31, 2018, in JAMA, the New England Journal of Medicine (NEJM), and The Lancet. Eligible studies were phase 3 and 4 trials with 1:1 randomization and statistically nonsignificant binary primary end points. Data analysis was performed from August 1, 2019, to August 31, 2019.

Exposures: Single vs multicenter enrollment, total number of events, private vs government funding, placebo vs active control, and time to event vs frequency data.

Main outcomes and measures: The primary outcome was the median RFI with interquartile range (IQR) at the P = .05 threshold. Secondary outcomes were the number of RCTs in which the number of participants lost to follow-up was greater than the RFI; the median RFI with IQR at different P value thresholds; the median reverse fragility quotient with IQR; and the correlation between sample sizes, number of events, and P values of the RCT and RFI.

Results: Of the 167 RCTs included, 76 (46%) were published in the NEJM, 50 (30%) in JAMA, and 41 (24%) in The Lancet. The median (IQR) sample size was 970 (470-3427) participants, and the median (IQR) number of events was 251 (105-570). The median (IQR) RFI at the P = .05 threshold was 8 (5-13). Fifty-seven RCTs (34%) had an RFI of 5 or lower, and in 68 RCTs (41%) the number of participants lost to follow-up was greater than the RFI. Trials with P values ranging from P = .06 to P = .10 had a median (IQR) RFI of 3 (2-4). When compared, median (IQR) RFIs were not statistically significant for single-center vs multicenter enrollment (5 [4-13] vs 8 [5-13]; P = .41), private vs government-funded studies (9 [5-13] vs 8 [5-13]; P = .34), and time-to-event primary end points vs frequency data (9 [5-14] vs 7 [4-13]; P = .43). The median (IQR) RFI at the P = .01 threshold was 12 (7-19) and at the P = .005 threshold was 14 (9-21).

Conclusions and relevance: This cross-sectional study found that a relatively small number of events (median of 8) had to change to move the primary end point of an RCT from nonsignificant to statistically significant. These findings emphasize the nuance required when interpreting trial results that did not meet prespecified significance thresholds.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Fonarow reported receiving personal fees from Abbott, AstraZeneca, Amgen, Bayer, Edwards, Janssen, Merck, and Medtronic outside the submitted work as well as being associate editor of JAMA Cardiology. Dr Friede reported receiving personal fees from Bayer, Novartis, Vifor, Enanta, Daiichy Sankyo, Johnson & Johnson, Boehringer Ingelheim, Roche, Fresenius Kabi, LivaNova, Galapagos, Penumbra, and Relaxera outside the submitted work. Dr Anker reported receiving grants and personal fees from Vifor and AV-Pharma as well as personal fees from Bayer, Boehringer Ingelheim, Novartis, Impulse Dynamics, and Servier outside the submitted work. Dr Butler reported receiving personal fees as a consultant from Abbott, Adrenomed, Amgen, Array, AstraZeneca, Bayer, Berlin Cures, Boehringer Ingelheim, Bristol-Myers Squib, CVRx, G3 Pharmaceutical, Innolife, Janssen, LinaNova, Luitpold, Medtronic, Merck, Novartis, Novo Nordisk, Relypsa, Roche, Sanofi, SC Pharma, V-Wave Limited, and Vifor outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Flowchart of Search Strategy
RCT indicates randomized clinical trial.
Figure 2.
Figure 2.. Scatterplot of the Correlation Between Reverse Fragility Index and P Value, Sample Size, and Total Number of Events
The size of the blue circles is proportional to the sample sizes. The size of the gray shading inside the blue circles is proportional to the number of events.
See this image and copyright information in PMC

References

    1. Alderson P. Absence of evidence is not evidence of absence. BMJ. 2004;328(7438):476-477. doi: 10.1136/bmj.328.7438.476 - DOI - PMC - PubMed
    1. Pocock SJ, McMurray JJV, Collier TJ. Making sense of statistics in clinical trial reports: part 1 of a 4-part series on statistics for clinical trials. J Am Coll Cardiol. 2015;66(22):2536-2549. doi: 10.1016/j.jacc.2015.10.014 - DOI - PubMed
    1. Wasserstein RL, Lazar NA. The ASA’s statement on P values: context, process, and purpose. Am Stat. 2016;70(2):129-133. doi: 10.1080/00031305.2016.1154108 - DOI
    1. Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial—survey of 71 negative trials. N Engl J Med. 1978;299(13):690-694. doi: 10.1056/NEJM197809282991304 - DOI - PubMed
    1. Akl EA, Briel M, You JJ, et al. Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ. 2012;344:e2809. doi: 10.1136/bmj.e2809 - DOI - PubMed