A tutorial on automatic post-stratification and weighting in conventional and regression-based norming of psychometric tests

Abstract

Norm scores are an essential source of information in individual diagnostics. Given the scope of the decisions this information may entail, establishing high-quality, representative norms is of tremendous importance in test construction. Representativeness is difficult to establish, though, especially with limited resources and when multiple stratification variables and their joint probabilities come into play. Sample stratification requires knowing which stratum an individual belongs to prior to data collection, but the required variables for the individual's classification, such as socio-economic status or demographic characteristics, are often collected within the survey or test data. Therefore, post-stratification techniques, like iterative proportional fitting (= raking), aim at simulating representativeness of normative samples and can thus enhance the overall quality of the norm scores. This tutorial describes the application of raking to normative samples, the calculation of weights, the application of these weights in percentile estimation, and the retrieval of continuous, regression-based norm models with the cNORM package on the R platform. We demonstrate this procedure using a large, non-representative dataset of vocabulary development in childhood and adolescence (N = 4542), using sex and ethnical background as stratification variables.

Keywords: Iterative proportional fitting; Post-stratification; Raking; Regression-based norming; Test construction.

Fig. 1

Continuous norm model based on weighted cases of the PPVT-IV dataset. Note. The plot depicts the manifest percentiles of the 15 distinct age groups (dots) and the fitted percentile curves (lines) for a selected set of percentiles ranging from PR 2.5 to PR 97.5. The curves are smooth and do not intersect, which is a requirement for a valid norm model