Prospective validation of a core curriculum progress assimilation instrument for radiation oncology residentship

Abstract

Objectives: To develop a tool that could assess residents' knowledge beyond simple information gathering and evaluate its reliability.

Methods: An assessment tool of 40 objective questions and at least one essay-based question was developed to assess residents' comprehension of general radiation oncology accordingly to validated training curricula beyond level 2 in the Bloom scale. With randomized content, questions were developed such as to be classified as at least 2 in the Bloom scale, so that reasoning and not only information gathering could be assessed. Criteria validation was made using the Classical Test Theory to describe difficulty and discrimination of each item. Reliability was tested by internal consistency using the Cronbach alpha test.

Results: Between 2016 and 2019, 24 residents were assessed. Six different versions of the test were made with a total of 240 objective questions and 18 essay-based questions. Five of the six versions were deemed valid and reliable. Comparisons between 1st (PGY-1) and 3rd (PGY-3) year residents were made. Consistently, PGY-3 residents had scores 150% higher than PGY-1 residents. Only two different PGY-3 reached the most complex level of answers in the essay-based questions. The results demonstrated that the major part of the acquired knowledge and retention occurs in the first six months of training rather than in all the following period.

Conclusion: The instrument can be considered valid. This developed instrument also raised the hypothesis that residents may not assess and analyze their acquired knowledge beyond the application level.

Keywords: Educational measurement; Graduate medical education; Radiation oncology; Residency.

Fig. 1

Variation of the R3/R1 scores by subject at each test version. The subjects selected were the ones with the most aberrant results. Subjects not represented in this figure: physics, radiobiology, general oncology and radiology had results close to breast; gastrointestinal, genitourinary and central nervous system reflected the same curve as head and neck (HN); thorax, skin, soft tissue sarcomas and benign conditions also had their results close to HN, but less reliable due to the limited number of items.