Development of a Summative Examination with Subject Matter Expert Validation

Abstract

Objective. To describe the development, implementation and impact of a summative examination on student learning and programmatic curricular outcomes. Methods. The summative examination was developed using a systematic approach. Item reliability was evaluated using standard psychometric analyses. Content validity was assessed using necessity scoring as determined by subject matter experts. Results. Almost 700 items written by 37 faculty members were evaluated. Passing standards increased annually (45% in 2009 to 67% in 2014) as the result of targeting item difficulty and necessity scores. The percentage of items exhibiting discrimination above 0.1 increased to 100% over the four years. Necessity scores above 2.75 out of 4 increased from 65% to 100% of items over six years of examination administration. Conclusion. This examination successfully assessed student and curricular outcomes. Faculty member engagement observed in this process supports a culture of assessment. This type of examination could be beneficial to other programs.

Keywords: assessment; examination development; subject matter experts; summative examination.

Figure 1.

Development and Administration of the Summative Examination.

Figure 2.

Passing Standard Calculations.

Figure 3.

Sample Student Report Card.

Figure 4.

Necessity Scores by Year. Subject matter experts’ second response to the following question for each item: To what extent is the knowledge, skill, or ability measured by this item necessary for minimally acceptable performance as a rising third-year student? (1=item is not necessary, 2=item is useful, 3=item is important, 4=item is essential or critical).