A Missing Data Approach to Correct for Direct and Indirect Range Restrictions with a Dichotomous Criterion: A Simulation Study

Abstract

A recurring methodological problem in the evaluation of the predictive validity of selection methods is that the values of the criterion variable are available for selected applicants only. This so-called range restriction problem causes biased population estimates. Correction methods for direct and indirect range restriction scenarios have widely studied for continuous criterion variables but not for dichotomous ones. The few existing approaches are inapplicable because they do not consider the unknown base rate of success. Hence, there is a lack of scientific research on suitable correction methods and the systematic analysis of their accuracies in the cases of a naturally or artificially dichotomous criterion. We aim to overcome this deficiency by viewing the range restriction problem as a missing data mechanism. We used multiple imputation by chained equations to generate complete criterion data before estimating the predictive validity and the base rate of success. Monte Carlo simulations were conducted to investigate the accuracy of the proposed correction in dependence of selection ratio, predictive validity, and base rate of success in an experimental design. In addition, we compared our proposed missing data approach with Thorndike's well-known correction formulas that have only been used in the case of continuous criterion variables so far. The results show that the missing data approach is more accurate in estimating the predictive validity than Thorndike's correction formulas. The accuracy of our proposed correction increases as the selection ratio and the correlation between predictor and criterion increase. Furthermore, the missing data approach provides a valid estimate of the unknown base rate of success. On the basis of our findings, we argue for the use of multiple imputation by chained equations in the evaluation of the predictive validity of selection methods when the criterion is dichotomous.

Fig 1. Missing data patterns under range restriction.

(a) Direct range restriction scenario (selection on X), and (b) indirect range restriction scenario (selection on Z). The shaded areas in Y represent the location of the missing values in the dataset.

Fig 2. Direct range restriction (DRR): Root mean square error (RMSE) of the estimates of the predictive validity (r^b and r^pb).

${\widehat{r}}_{\text{b}}$ is the estimate of the biserial correlation coefficient for an artificially dichotomous criterion variable, and ${\widehat{r}}_{\text{pb}}$ is the estimate of the point-biserial correlation coefficient for a naturally dichotomous criterion variable.

Fig 3. Indirect range restriction (IRR): Root mean square error (RMSE) of the estimates of the predictive validity (r^b and r^pb).

${\widehat{r}}_{\text{b}}$ is the estimate of the biserial correlation coefficient for an artificially dichotomous criterion variable, and ${\widehat{r}}_{\text{pb}}$ is the estimate of the point-biserial correlation coefficient for a naturally dichotomous criterion variable.

Fig 4. Direct range restriction (DRR): Effects of a weak, moderate, and strong predictive validity on the root mean square error (RMSE) of the estimates of the predictive validity (r^b and r^pb).

${\widehat{r}}_{\text{b}}$ is the estimate of the biserial correlation coefficient for an artificially dichotomous criterion variable, and ${\widehat{r}}_{\text{pb}}$ is the estimate of the point-biserial correlation coefficient for a naturally dichotomous criterion variable.

Fig 5. Indirect range restriction (IRR): Effects of a weak, moderate, and strong predictive validity on the root mean square error (RMSE) of the estimates of the predictive validity (r^b and r^pb).

${\widehat{r}}_{\text{b}}$ is the estimate of the biserial correlation coefficient for an artificially dichotomous criterion variable, and ${\widehat{r}}_{\text{pb}}$ is the estimate of the point-biserial correlation coefficient for a naturally dichotomous criterion variable.

Fig 6. Indirect range restriction (IRR): Effects of a weak, moderate, and strong relationship between predictor X and selection variable Z on the root mean square error (RMSE) of the estimates of the predictive validity (r^b and r^pb).

${\widehat{r}}_{\text{b}}$ is the estimate of the biserial correlation coefficient for an artificially dichotomous criterion variable, and ${\widehat{r}}_{\text{pb}}$ is the estimate of the point-biserial correlation coefficient for a naturally dichotomous criterion variable.