Power Calculations to Select Instruments for Clinical Trial Secondary Endpoints. A Case Study of Instrument Selection for Post-Traumatic Stress Symptoms in Subjects with Acute Respiratory Distress Syndrome

Abstract

Rationale: After the sample size of a randomized clinical trial (RCT) is set by the power requirement of its primary endpoint, investigators select secondary endpoints while unable to further adjust sample size. How the sensitivity and specificity of an instrument used to measure these outcomes, together with their expected underlying event rates, affect an RCT's power to measure significant differences in these outcomes is poorly understood.

Objectives: Motivated by the design of an RCT of neuromuscular blockade in acute respiratory distress syndrome, we examined how power to detect a difference in secondary endpoints varies with the sensitivity and specificity of the instrument used to measure such outcomes.

Methods: We derived a general formula and Stata code for calculating an RCT's power to detect differences in binary outcomes when such outcomes are measured with imperfect sensitivity and specificity. The formula informed the choice of instrument for measuring post-traumatic stress-like symptoms in the Reevaluation of Systemic Early Neuromuscular Blockade RCT ( www.clinicaltrials.gov identifier NCT02509078).

Measurements and main results: On the basis of published sensitivities and specificities, the Impact of Events Scale-Revised was predicted to measure a 36% symptom rate, whereas the Post-Traumatic Stress Symptoms instrument was predicted to measure a 23% rate, if the true underlying rate of post-traumatic stress symptoms were 25%. Despite its lower sensitivity, the briefer Post-Traumatic Stress Symptoms instrument provided superior power to detect a difference in rates between trial arms, owing to its higher specificity.

Conclusions: Examining instruments' power to detect differences in outcomes may guide their selection when multiple instruments exist, each with different sensitivities and specificities.

Keywords: bias; clinical trials; critical care outcomes; sensitivity; specificity.

Figure 1.

Comparison of the ability of the Post-Traumatic Stress Symptoms instrument (PTSS-14) and the Impact of Events Scale-Revised (IES-R) to detect a difference in post-traumatic stress–like symptoms between study arms. The PTSS-14 is a 14-item instrument with a reported sensitivity of 86% and specificity of 97% for detecting post-traumatic stress–like symptoms in patients recovering from critical illness. The IES-R is a 22-item instrument with a reported sensitivity of 100% and specificity of 85% for detecting post-traumatic stress–like symptoms in patients recovering from critical illness. A 25% baseline symptom rate was assumed in the control arm, and an increased level was assumed in the treatment arm. Power calculations assume a 500-patient sample size in each arm; the true test characteristics of each instrument were assumed to be the reported point estimates of the sensitivity and specificity of each instrument in its validation studies, and test performance would be similar over the course of the trial.

Figure 2.

Comparison of the ability of the Post-Traumatic Stress Symptoms instrument (PTSS-14) and the Impact of Events Scale-Revised (IES-R) to detect a difference in post-traumatic stress–like symptoms between study arms when the IES-R’s threshold is raised to improve its specificity. The PTSS-14 is a 14-item instrument with a reported sensitivity of 86% and specificity of 97% for detecting post-traumatic stress–like symptoms in patients recovering from critical illness. The IES-R is a 22-item instrument with a reported sensitivity of 75% and specificity of 92% in intensive care unit patients for detecting post-traumatic stress–like symptoms when using a higher diagnostic threshold. A 25% baseline symptom rate was assumed in the control arm, and an increased level was assumed in the treatment arm. Power calculations assume a 500-patient sample size in each arm; the true test characteristics of each instruments were assumed to be the reported point estimates of the sensitivity and specificity of each instrument in its validation studies, and test performance would be similar over the course of the trial.

Figure 3.

Statistical power to measure a 20% absolute difference in the rate of an outcome across varying instrument sensitivity and specificity, and underlying outcome prevalence. (A) Outcome rate of 10% in control group and 30% in treatment group, (B) outcome rate of 25% in control group and 45% in treatment group, and (C) outcome rate of 50% in control group and 70% in treatment group.

Figure 3.

Statistical power to measure a 20% absolute difference in the rate of an outcome across varying instrument sensitivity and specificity, and underlying outcome prevalence. (A) Outcome rate of 10% in control group and 30% in treatment group, (B) outcome rate of 25% in control group and 45% in treatment group, and (C) outcome rate of 50% in control group and 70% in treatment group.