Development and Validation of the Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS)

Abstract

Importance: A new outcome measure for overall disability level with improved responsiveness is needed for amyotrophic lateral sclerosis (ALS) clinical trials.

Objective: To describe the creation and development of a new self-reported ALS disability scale with improved item targeting and psychometric properties that used a mathematically rigorous Rasch methodology.

Design, setting, and participants: A preliminary ALS disability questionnaire with 119 questions was created based on literature review, clinical judgement of an expert panel, and patient input. Patients with ALS were recruited from January 2017 to June 2019 from the Emory University and Atlanta VA Medical Center ALS clinics, both in Atlanta, Georgia, during regularly scheduled clinic appointments to complete the draft questionnaire and standard ALS outcome measures. All consecutive patients seen at the Emory University and Atlanta VA Medical Center ALS clinics during the recruitment period with a diagnosis of ALS who were able to provide informed consent were invited to participate in the study. Rasch analyses were performed, and items were systematically removed based on missing data, model fit, disordered thresholds, item bias, and clinical judgment. A total of 509 patients with ALS were seen at the 2 sites during the recruitment period, and 264 patients provided informed consent.

Interventions: Participants completed the draft Rasch questionnaire and the revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R).

Main outcomes and measures: Rasch analyses and standard scale metrics were performed to create the new scale, and Rasch analyses were performed on the ALSFRS-R for comparison.

Results: Overall, 243 participants with ALS completed the draft questionnaire, and 230 participants were included for Rasch analyses. The mean (SD) age for study participants was 61.9 (11.1) years, 146 (60.1%) were men, and site of onset was 23.0% bulbar (n = 56), 36.2% upper extremity (n = 88), and 39.5% lower extremity (n = 96). A 28-question Rasch-Built Overall ALS Disability Scale (ROADS) was constructed with each item scored 0, 1, or 2. The ROADS fulfilled Rasch model requirements, demonstrated improved item targeting compared with the ALSFRS-R, and had test-retest reliability of 0.97. Individual question fit statistics demonstrated infit values from 0.68 to 1.37 and outfit values from 0.66 to 1.43. The difference between the empirical variance explained by the measures and the modeled variance was 0.1%. The ALSFRS-R violated Rasch model expectations and demonstrated disordered thresholds for 9 of 12 questions; 13 of 48 answer choices on the ALSFRS-R were never the most probable answer choice for any overall disability level.

Conclusions and relevance: In this study, the 28-question, self-reported ROADS, which is linearly weighted, had improved item targeting compared with the ALSFRS-R, had high test-retest reliability, and was validated. ROADS may serve as a valuable and easily accessible outcome measure for use in ALS trials and in the clinic with improved responsiveness compared with the ALSFRS-R.

Figure 1.. Category Probability Curves

A, Using a knife and fork was included in the Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS) with appropriately ordered category probability curves. Category 2 is the most probable answer choice for patients with the least amount of overall disability. As overall disability increases, category 1 sequentially becomes the most probable answer choice for patients with moderate overall disability level. As overall disability increases to the greatest level, 0 sequentially becomes the most probable answer choice. B, Driving a car was removed from ROADS owing to disordered thresholds. Category 2 is the most probable answer choice for patients with the least amount of overall disability, but category 0 appears out of order as the most probable answer choice for patients with a higher level of overall disability, and category 1 is never the most probable answer choice for any overall disability level. C, An example of disordered thresholds from the revised Amyotrophic Lateral Sclerosis Functional Rating Scale. Item responses 1, 2, and 3 are never the most probable answer choice for any overall functional rating ability.

Figure 2.. Threshold Map and Rasch Measures

The Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale threshold map shows the difficulty order and item targeting for each question, with the most difficult task to perform (get heavy objects off a high shelf) appearing on the top and the easiest task to perform (nod yes or no) appearing on the bottom. The x-axis represents the logit measure for overall disability level. The placement of the 0, 1, and 2 responses for each item demonstrate the most probable answer at any particular overall disability level, with the open circles representing the transition point where a 0 and 1 response or a 1 and 2 response are equally probable. The threshold map shows a gradual shift from mostly 2 answers on the right for patients with the least disabilities to more 1 and 0 answers as overall disability level declines. Each item was scored as follows: 0, unable to perform; 1, abnormal: able to perform but with difficulty; or 2, normal: able to perform without difficulty.

Figure 3.. Improved Item Targeting on the ROADS Compared With the ALSFRS-R

The Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS) targets a broader range of ability levels compared with the revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R). Each questionnaire item is placed in difficulty order with the most difficult items on top and the easiest items at the bottom, and the measure column indicates the logit measure where the middle item response choice is the most probable answer. Item difficulties spanned 5.9 logits for the ROADS (range, −2.97 to 2.93 logits) but only 2.27 logits (range, −1.32 to 0.95 logits) for the ALSFRS-R.