Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: the GAIN substance problem scale

Abstract

Substance abuse treatment programs are being pressed to measure and make clinical decisions more efficiently about an increasing array of problems. This computerized adaptive testing (CAT) simulation examined the relative efficiency, precision and construct validity of different starting and stopping rules used to shorten the Global Appraisal of Individual Needs' (GAIN) Substance Problem Scale (SPS) and facilitate diagnosis based on it. Data came from 1,048 adolescents and adults referred to substance abuse treatment centers in 5 sites. CAT performance was evaluated using: (1) average standard errors, (2) average number of items, (3) bias in person measures, (4) root mean squared error of person measures, (5) Cohen's kappa to evaluate CAT classification compared to clinical classification, (6) correlation between CAT and full-scale measures, and (7) construct validity of CAT classification vs. clinical classification using correlations with five theoretically associated instruments. Results supported both CAT efficiency and validity.

Appendix A

Flow chart of CAT Procedure Using Variable Stop Rules and the SPS Screener to Set the Start Valu

Figure 1

Rasch Based Substance Problem Scale Severity (x-axis) by the prevalence of past year abuse/dependence (stacked area) with the mid point ranges for triage groups focused on overall abuse/dependence (general severity) and just dependence (high severity).

Figure 2

Illustration of the Effect of Screener Based Starting Rule and Variable Stopping Rule on Standard Error of CAT-Estimated SDS Measures Across the SDS Continuum (screener-based estimated starting value, Low SE=0.75, Mid SE=0.35, High SE=0.75)