Indices of non-ignorable selection bias for proportions estimated from non-probability samples

Abstract

Rising costs of survey data collection and declining response rates have caused researchers to turn to non-probability samples to make descriptive statements about populations. However, unlike probability samples, non-probability samples may produce severely biased descriptive estimates due to selection bias. The paper develops and evaluates a simple model-based index of the potential selection bias in estimates of population proportions due to non-ignorable selection mechanisms. The index depends on an inestimable parameter ranging from 0 to 1 that captures the amount of deviation from selection at random and is thus well suited to a sensitivity analysis. We describe modified maximum likelihood and Bayesian estimation approaches and provide new and easy-to-use R functions for their implementation. We use simulation studies to evaluate the ability of the proposed index to reflect selection bias in non-probability samples and show how the index outperforms a previously proposed index that relies on an underlying normality assumption. We demonstrate the use of the index in practice with real data from the National Survey of Family Growth.

Keywords: Non-ignorable selection bias; Non-probablity samples; Selection at random; Survey data collection.

Fig. 1.

MUBP(ϕ) from the probit model and MUB(ϕ) from the normal model versus the true estimated bias, shown for combinations of the biserial correlation corr(U, X)=ρ_ux (rows) and the selection mechanism (columns), for E[Y]=0.3 (results are medians across 1000 simulated data sets for each scenario): ●, probit, MUBP(0); ▲, probit, MUBP(0.5); ■, probit, MUBP(1); ○, normal, MUBP(0); △, normal, MUBP(0.5); □, normal, MUBP(1); – – –, equality (index=estimated bias)

Fig. 2.

Correlation between MUBP(ϕ) and true estimated bias, and between MUB(ϕ) and true estimated bias, versus the biserial correlation corr(U, X)=ρ_ux, for combinations of selection mechanism (columns), μ_Y (rows) and ϕ (shape) (results from all estimated biases (all values of β_Z and β_U) are all plotted together; correlations are estimated from 1000 simulated data sets for each scenario): ●, probit, MUBP(0); ▲, probit, MUBP(0.5); ■, probit, MUBP(1); ○, normal, MUBP(0); △, normal, MUBP(0.5); □, normal, MUBP(1)

Fig. 3.

Coverage of [MUBP(0), MUBP(1)] and [SMUB(0), SMUB(1)] ML or MML intervals, and Bayesian credible intervals, shown as a function of the true estimated bias (x-axis), selection mechanism and estimation method (columns), proxy strength (rows) and E[Y] (shape) (coverages are estimated from 1000 simulated data sets): ●, normal–ML; ●, probit–MML; ■, normal–Bayes; ■, probit–Bayes; ●, E[Y]=0.1; ▲, E[Y]=0.3; ■, E[Y]=0.5