Measures of the Degree of Departure from Ignorable Sample Selection

Abstract

With the current focus of survey researchers on "big data" that are not selected by probability sampling, measures of the degree of potential sampling bias arising from this nonrandom selection are sorely needed. Existing indices of this degree of departure from probability sampling, like the R-indicator, are based on functions of the propensity of inclusion in the sample, estimated by modeling the inclusion probability as a function of auxiliary variables. These methods are agnostic about the relationship between the inclusion probability and survey outcomes, which is a crucial feature of the problem. We propose a simple index of degree of departure from ignorable sample selection that corrects this deficiency, which we call the standardized measure of unadjusted bias (SMUB). The index is based on normal pattern-mixture models for nonresponse applied to this sample selection problem and is grounded in the model-based framework of nonignorable selection first proposed in the context of nonresponse by Don Rubin in 1976. The index depends on an inestimable parameter that measures the deviation from selection at random, which ranges between the values zero and one. We propose the use of a central value of this parameter, 0.5, for computing a point index, and computing the values of SMUB at zero and one to provide a range of the index in a sensitivity analysis. We also provide a fully Bayesian approach for computing credible intervals for the SMUB, reflecting uncertainty in the values of all of the input parameters. The proposed methods have been implemented in R and are illustrated using real data from the National Survey of Family Growth.

Keywords: Measures of selection bias; National Survey of Family Growth; Nonignorable sample selection; Nonprobability sampling; Sampling bias.

Figure 1.

Scatterplot of STEB Against SMUB(0), Including Measures of Linear Association. note: rho =$r_{\mathit{XY}}^{(1)}$.

Figure 2.

Scatterplot of STEB Against SMUB(0.5), Including Measures of Linear Association. note: rho =$r_{\mathit{XY}}^{(1)}$.

Figure 3.

Scatterplot of STEB Against SMUB(1), Including Measures of Linear Association. note: rho =$r_{\mathit{XY}}^{(1)}$.

Figure 4.

Scatterplot of STEB Against FMI, Including Measures of Linear Association. note: rho =$r_{\mathit{XY}}^{(1)}$.

Figure 5.

Scatterplot of Drawn Values of SMUB Versus Drawn Values of the $\phi$ Parameter for the Mean of Number of Months Worked in the Past Twelve Months (Females), Following Our Proposed Bayesian Approach in the Presence of Sufficient Statistics on Z for Nonsampled Cases.