Correcting the predictive validity of a selection test for the effect of indirect range restriction

Abstract

Background: The validity of selection tests is underestimated if it is determined by simply calculating the predictor-outcome correlation found in the admitted group. This correlation is usually attenuated by two factors: (1) the combination of selection variables which can compensate for each other and (2) range restriction in predictor and outcome due to the absence of outcome measures for rejected applicants.

Methods: Here we demonstrate the logic of these artifacts in a situation typical for student selection tests and compare four different methods for their correction: two formulas for the correction of direct and indirect range restriction, expectation maximization algorithm (EM) and multiple imputation by chained equations (MICE). First we show with simulated data how a realistic estimation of predictive validity could be achieved; second we apply the same methods to empirical data from one medical school.

Results: The results of the four methods are very similar except for the direct range restriction formula which underestimated validity.

Conclusion: For practical purposes Thorndike's case C formula is a relatively straightforward solution to the range restriction problem, provided distributional assumptions are met. With EM and MICE more precision is obtained when distributional requirements are not met, but access to a sophisticated statistical package such as R is needed. The use of true score correlation has its own problems and does not seem to provide a better correction than other methods.

Keywords: EM; Mice; Predictive validity; Range restriction; Student selection; Suppression.

Fig. 1

a Scattergram of X₁ and X₂. 20% of 1000 applicants are selected by the sum of X₁ and X₂; the circular cloud representing all applicants is divided by a diagonal line that separates the top right area from the bottom left area. b This generates a negative correlation between X₁ and X₂ in the incumbents $\left(r_{x_1{x}_2\mid i}=-0.71\right)$. Residuals of X₁ after the linear effect of X₂ is removed. They are expressed as deviations from the regression line: The residuum of X₁ when the influence of X₂ is removed is the observed X₁ value minus the expected value of the regression X₁ on X₂

Fig. 2

Relations between X₁, X₂, and Y in applicants and incumbents for 20% selection rate. $r_{y{x}_1}$: first order correlation, ${\beta }_{y{x}_1}$: beta coefficient, $r_{{\mathit{yx}}_{1•2}}$: semipartial correlation

Fig. 3

The correlation in the full sample of applicants (a) is larger than the correlation in the incumbents (b) due to range restriction: The variances of X₁, X₂ and Y are restricted

Fig. 4

a-c Scattergram of Y (study success) with X₁ (test results) and precision of the estimation of $r_{{\mathit{yx}}_1\mid a}$ (predictive validity) from different methods when 30%, 20%, and 10% of applicants are selected

Fig. 5

Scattergram of HAM-Nat and hGPA at Hamburg medical school for the 2011 cohort

Fig. 6

Relations between HAM-Nat, hGPA, and Study Success in the incumbents. $r_{y{x}_1}$: first order correlation, ${\beta }_{y{x}_1}$: beta coefficient, $r_{{\mathit{yx}}_{1•2}}$: semipartial correlation