Evaluating the impact of calibration of patient-reported outcomes measures on results from randomized clinical trials: a simulation study based on Rasch measurement theory

Abstract

Background: Meaningfully interpreting patient-reported outcomes (PRO) results from randomized clinical trials requires that the PRO scores obtained in the trial have the same meaning across patients and previous applications of the PRO instrument. Calibration of PRO instruments warrants this property. In the Rasch measurement theory (RMT) framework, calibration is performed by fixing the item parameter estimates when measuring the targeted concept for each individual of the trial. The item parameter estimates used for this purpose are typically obtained from a previous "calibration" study. But imposing this constraint on item parameters, instead of freely estimating them directly in the specific sample of the trial, may hamper the ability to detect a treatment effect. The objective of this simulation study was to explore the potential negative impact of calibration of PRO instruments that were developed using RMT on the comparison of results between treatment groups, using different analysis methods.

Methods: PRO results were simulated following a polytomous Rasch model, for a calibration and a trial sample. Scenarios included varying sample sizes, with instrument of varying number of items and modalities, and varying item parameters distributions. Different treatment effect sizes and distributions of the two patient samples were also explored. Cross-sectional comparison of treatment groups was performed using different methods based on a random effect Rasch model. Calibrated and non-calibrated approaches were compared based on type-I error, power, bias, and variance of the estimates for the difference between groups.

Results: There was no impact of the calibration approach on type-I error, power, bias, and dispersion of the estimates. Among other findings, mistargeting between the PRO instrument and patients from the trial sample (regarding the level of measured concept) resulted in a lower power and higher position bias than appropriate targeting.

Conclusions: Calibration does not compromise the ability to accurately assess a treatment effect using a PRO instrument developed within the RMT paradigm in randomized clinical trials. Thus, given its essential role in producing interpretable results, calibration should always be performed when using a PRO instrument developed using RMT as an endpoint in a randomized clinical trial.

Keywords: Calibration; Clinical trials; Patient-reported outcomes; Rasch measurement theory.

Fig. 1

Illustration of the archetypes of items distribution, for different scenarios. Legend: Vertical dashed lines represent the item response category thresholds (δ_jl, with each color corresponding to a different item) in different scenarios, and the probability density function curve represents the distribution of the latent trait in the calibration sample (case with a variance = 1). The left part of the figure includes cases where the item locations δ_j have a low dispersion (range = 0.5) and the δ_jl have a high dispersion (SD = 2.5). The right part of the figure includes cases where the item locations δ_j have a high dispersion (range = 2) and the δ_jl have a low dispersion (SD = 1.5). Each line corresponds to different scenarios regarding the number of item and modalities: A) J = 4 items, M = 3 modalities. B) J = 4 items, M = 5 modalities. C) J = 10 items, M = 5 modalities. Full values for the response category thresholds δ_jl are provided in supplementary materials (Additional file 1)

Fig. 2

Power using calibrated and non-calibrated approaches, depending on mistargeting of the trial sample μ. Legend: power is presented for instruments with varying number of items J and modalities M. Presented results are for comparison of treatment groups based on $\widehat{\gamma }$, for scenarios with the distribution of the item parameters = second archetype with SD of 1.5 and range of 2, γ = 0.2, N_trial = 500, N_calibration = 250, variance of the calibration sample = 1