Psychometric properties of the International Society of Wheelchair Professionals' basic manual wheelchair-service-provision knowledge Test Version 1 and development of Version 2

Abstract

Introduction: Valid and reliable scores from measurement tools to test competency in basic manual wheelchair-service-provision are needed to promote good practice and support capacity building. The International Society of Wheelchair Professionals' (ISWP) Basic Test Version 1 in English, launched in 2015, is the most frequently used outcome measure tool to test basic manual wheelchair-service-provision knowledge and is part of an international certification process. Despite the wide acceptance and use of the test, its psychometric properties have not yet been established. The objectives of this study were 1) to evaluate the test's psychometric properties, 2) to develop the test's Version 2, and 3) to evaluate the content validity of the new version.

Methods: For Objective 1, methods from the Classical Test Theory were used to obtain items' difficulty, item discrimination index and domains' reliability. For Objective 2, a team of experts in wheelchair service delivery and education conducted a systematic qualitative review of the questions' text and answers and updated them using evidence-based guidelines. For Objective 3, an external team reviewed the clarity, relevance and domain allocation of the developed items using a 4-point Likert scale. Descriptive statistics were used to describe and characterize the results for each objective. Item-content (I-CVI) and Scale-content (S-CVI) validity indexes were calculated to compute content validity.

Results: For Objective 1, all domains in the test were below the threshold for acceptable internal consistency reliability; 80% of the total test pool (116 items from the total pool of 145) did not meet the thresholds for item difficulty and index of discrimination suggested in the literature. Of the items in the Test, 78% could be responded to intuitively and 66% did not distinguish between test-takers who were knowledgeable in the content area and those who were not. For Objective 2, experts found concerns such as items being grouped in the wrong domain, being repeated, not using person-first language, and using terms inconsistently. Thirty-four (23.4%) items were dropped and 111 (76.5%) were updated. In addition, 61 new items were developed. Members re-categorized the items and proposed a new classification of subdomains. For Objective 3, good agreement between subject-matter experts was found; the S-CVI calculated using the I-CVIs related to item clarity was 84% while using the I-CVIs related to item relevance was 98%. Only 7 items (4.1%) were deemed to be in the wrong domain and 4 items (2.3%) were considered irrelevant and dropped.

Conclusion: The psychometric evidence in support of ISWP Basic Test Version 1 in English is suboptimal. A new set of items developed by experts in the field has shown excellent content validity. Ongoing assessments will be needed as ISWP Basic Test Version 2 is implemented and monitored.