Item response theory-based measure of global disability in multiple sclerosis derived from the Performance Scales and related items

Abstract

Background: The eight Performance Scales and three assimilated scales (PS) used in North American Research Committee on Multiple Sclerosis (NARCOMS) registry surveys cover a broad range of neurologic domains commonly affected by multiple sclerosis (mobility, hand function, vision, fatigue, cognition, bladder/bowel, sensory, spasticity, pain, depression, and tremor/coordination). Each scale consists of a single 6-to-7-point Likert item with response categories ranging from "normal" to "total disability". Relatively little is known about the performances of the summary index of disability derived from these scales (the Performance Scales Sum or PSS). In this study, we demonstrate the value of a combination of classical and modern methods recently proposed by the Patient-Reported Outcome Measurement Information System (PROMIS) network to evaluate the psychometric properties of the PSS and derive an improved measure of global disability from the PS.

Methods: The study sample included 7,851adults with MS who completed a NARCOMS intake questionnaire between 2003 and 2011. Factor analysis, bifactor modeling, and item response theory (IRT) analysis were used to evaluate the dimension(s) of disability underlying the PS; calibrate the 11 scales; and generate three alternative summary scores of global disability corresponding to different model assumptions and practical priorities. The construct validity of the three scores was compared by examining the magnitude of their associations with participant's background characteristics, including unemployment.

Results: We derived structurally valid measures of global disability from the PS through the proposed methodology that were superior to the PSS. The measure most applicable to clinical practice gives similar weight to physical and mental disability. Overall reliability of the new measure is acceptable for individual comparisons (0.87). Higher scores of global disability were significantly associated with older age at assessment, longer disease duration, male gender, Native-American ethnicity, not receiving disease modifying therapy, unemployment, and higher scores on the Patient Determined Disease Steps (PDDS).

Conclusion: Promising, interpretable and easily-obtainable IRT scores of global disability were generated from the PS by using a sequence of traditional and modern psychometric methods based on PROMIS recommendations. Our analyses shed new light on the construct of global disability in MS.

Figure 1

Unidimensional CFA model of self-assessed neurological disability in NARCOMS registrants. Note: “Disability” represents a latent factor, i.e., a not directly observable continuous variable whose scale is inferred from the variability and correlations among PS. “PS-factor correlations” are estimates of the correlations between PS and factor scores. “Residual variances” represent the fractions of PS score variability that are not explained by the factor.

Figure 2

Bifactor CFA model of self-assessed neurological disability in NARCOMS registrants. Note: “Global” represents the general factor of global disability; “Physical” and “Mental” represent the auxiliary factors of “physical” and “mental” disability. Correlations among the three factors are all forced to be zero. Thus, the physical and mental factors each explain a fraction of the variability in PS scores left unexplained by the general factor. Comparisons of “Residual variances” in Figure 2 and Figure 1, provide information about the fraction of variability in PS scores that the two auxiliary factors explain above and beyond the general factor.

Figure 3

Relation among Performance Scale (PS) categories, Performance Scales Sum (PSS-11) scores and PS IRT scores of global disability (from unidimensional model) in a sample of NARCOMS registrants.

Figure 4

Relations among the levels of the Performance Scales Sum (PSS-11) score and the IRT summed score of global disability. Note: To facilitate comparisons, the levels of the IRT summed score were linearly transformed to range from 0 to 46 (i.e., the range of the PSS-11 levels). Thus, had the levels of the two scores been perfectly equivalent, then the dots on the figure would have been aligned on the diagonal line.

Figure 5

Associations between Performance Scales-based, IRT-derived, scores of disability and PDDS scores in a sample of NARCOMS registrants. a) Scores of global disability generated from the bifactor IRT model (●) versus the unidimensional IRT model (□, pattern scores; ▲, summed scores). b) Scores of residual physical disability generated from the bifactor IRT model. c) Scores of residual mental disability generated from the bifactor IRT model. Note: Although the scores of global disability, residual physical disability, and residual mental disability are all reported as scaled scores (Mean, 50; SD, 15), they are not on the same metric. Error bars represent 95% confidence intervals. Spearman correlations between IRT scores and PDDS scores: Figure 5a) ● 0.60, □ 0.65, ▲0.68; Figure 5b) 0.72; Figure 5c) 0.07.

Figure 6

Relations between distributions of Performance Scales IRT scores (left) and raw Performance Scales Sum (PSS-11) scores (right) among employed and unemployed NARCOMS registrants. Note: Data on causes of unemployment were not available.