B-value and empirical equivalence bound: A new procedure of hypothesis testing

Abstract

In this study, we propose a two-stage procedure for hypothesis testing, where the first stage is conventional hypothesis testing and the second is an equivalence testing procedure using an introduced empirical equivalence bound (EEB). In 2016, the American Statistical Association released a policy statement on P-values to clarify the proper use and interpretation in response to the criticism of reproducibility and replicability in scientific findings. A recent solution to improve reproducibility and transparency in statistical hypothesis testing is to integrate P-values (or confidence intervals) with practical or scientific significance. Similar ideas have been proposed via the equivalence test, where the goal is to infer equality under a presumption (null) of inequality of parameters. However, the definition of scientific significance/equivalence can sometimes be ill-justified and subjective. To circumvent this drawback, we introduce the B-value and the EEB, which are both estimated from the data. Performing a second-stage equivalence test, our procedure offers an opportunity to improve the reproducibility of findings across studies.

Keywords: empirical equivalence bound; equivalence test; hypothesis testing.

FIGURE 1

Conclusion of an equivalence test with different prespecified equivalence bounds when (a) the confidence interval covers zero and (b) the confidence interval does not cover zero. Two possible conclusions under scenario (a): (i) inconclusive result and (ii) equivalence. Three possible conclusions under scenario (b): (i) significant difference, (ii) inconclusive result, and (iii) equivalence.

FIGURE 2

A two-stage testing procedure comparing two means.

FIGURE 3

Marginal and conditional distribution of the B-value and the corresponding EEB at various β levels in the example.

FIGURE 4

(a) The marginal and conditional distribution of the B-value and (b) the marginal and conditional EEB at various β levels in the example.

FIGURE 5

Implementation of the two-stage testing procedure on the example data with α = 0.05 and β = 0.95. The black solid lines in Step 1 are the 95% confidence intervals, the black solid lines in Step 2 are the 90% confidence intervals, and the gray dashed lines are the equivalence intervals with margin level at EEB_α(β | C).

FIGURE 6

(a)&(c) Data scatter plot and the 95% confidence interval (CI) of each species. (b)&(d) The conditional empirical equivalence interval with β = 0.95 and the 90% CI of each pair-wise comparison.

FIGURE 7

(a) Scatter plot and 95% confidence interval (CI) of the performance data after adjusting for age, sex, and motor coordination. (b) The conditional empirical equivalence interval with β = 0.95 and the 90% CI of the comparison at each repetition.