Rasch Analysis for Instrument Development: Why, When, and How?

Abstract

This essay describes Rasch analysis psychometric techniques and how such techniques can be used by life sciences education researchers to guide the development and use of surveys and tests. Specifically, Rasch techniques can be used to document and evaluate the measurement functioning of such instruments. Rasch techniques also allow researchers to construct "Wright maps" to explain the meaning of a test score or survey score and develop alternative forms of tests and surveys. Rasch techniques provide a mechanism by which the quality of life sciences-related tests and surveys can be optimized and the techniques can be used to provide a context (e.g., what topics a student has mastered) when explaining test and survey results.

FIGURE 1.

Thinking about linear measurement. A meter stick being used to make linear measures and compare the height of three flowers.

FIGURE 2.

Example test scores. The raw test scores of four ninth-grade students who completed the same 25-item test. Twenty items were appropriate for ninth-grade students, but five test items were at college level.

FIGURE 3.

Example survey rating scale. For the Q#5 scale, the “jump” between each of the ratings is equal. For the second (Q#8) and third (Q#10) scales, the “jump” from each rating to the next rating is not equal. Furthermore, the way the rating scale functions across the items is not identical. All that a researcher can assert is that the rating scale is ordinal (SA > A > D > SD) for each item.

FIGURE 4.

Rasch measurement schematic. To measure, an analyst must 1) consider a single construct (represented by the vertical line); 2) consider the parts of the variable marked by different test items; 3) understand that a test taker will be located at some point along the variable; and 4) understand that the probability of a respondent answering a test item correctly can be expressed.

FIGURE 5.

The dichotomous Rasch model. B_n is the ability of the test taker along the variable; D_i is the difficulty of a test item; P_ni is the probability of the test taker correctly answering a specific test item; and 1 − P_ni is the probability of a test taker incorrectly answering a test item.

FIGURE 6.

Example Wright map. A Wright map can allow researchers to quickly identify strengths and weaknesses of an instrument. For example, are some test items measuring the same part of the variable? Are there portions of the tested variable that are missing test items? Investigating the location and distribution of test items on a Wright map is akin to reviewing the marks placed on a meter stick.

FIGURE 7.

Multiple test forms. An example of how item anchors can be used to link the measurement scale of different test forms. Four items (4, 5, 6, and 7) are common to forms A and B, allowing the two scales to be linked. Four items (18, 19, 20, and 21) are common to forms B and C, allowing all the test takers (regardless of the form completed) to be expressed on the same scale.

FIGURE 8.

Making inferences using Wright maps. A Wright map allows a researcher to explain the meaning of the growth observed from pre to post. The items falling between the pre and post lines help describe the growth.