Computerized adaptive testing of symptom severity: a registry-based study of 924 patients with trapeziometacarpal arthritis

Abstract

We aimed to develop a computerized adaptive testing (CAT) version of the 11 item Patient Evaluation Measure (PEM), using an item response theory model. This model transformed the ordinal scores into ratio-interval scores. We obtained PEM responses from 924 patients with trapeziometacarpal osteoarthritis to build a CAT model and tested its performance on a simulated cohort of 1000 PEM response sets. The CAT achieved high precision (median standard error or measurement 0.26) and reduced the number of questions needed for accurate scoring from 11 to median two. The CAT scores and item-response-theory-based 15-item PEM scores were similar, and a Bland-Altman analysis demonstrated a mean score difference of 0.2 between the CAT and the full-length PEM scores on a scale from 0 to 100. We conclude that the CAT substantially reduced the burden of the PEM while also harnessing the validity of item response theory scoring.

Keywords: CAT; Osteoarthritis; PROM; computerized adaptive testing; patient-reported outcome measures.

Figure 1.

Example of change in score estimation and error for an individual completing the CAT. The score is on the y-axis. The person’s true score is 57/100 (red). The current estimate of the person’s score is the black line. As they complete the CAT by first responding to question 8 and thereafter to question 3, the score estimate from the CAT improves, with error (SEm) shown as error bars.

Figure 2.

Comparison of CAT, which typically used between 1 and 4 items per individual, to IRT-based PEM scores using all 11 items.

Figure 3.

Bland–Altman plot comparing CAT performance with full-length PEM. The mean of the CAT and full-length scores for each individual is on the x-axis. The difference between the individuals’ scores is on the y-axis. The solid horizontal line shows the mean difference for all scores, and the dashed horizontal lines show the 95% limits of agreement. Increasing intensity in colour of the dots represents multiple data points of the same value overlapping.