Rubric vs. numeric rating scale: agreement among evaluators on endodontic treatments performed by dental students

Abstract

Background: Students´ assessment should be carried out in an effective and objective manner, which reduces the possibility of different evaluators giving different scores, thus influencing the qualification obtained and the consistency of education. The aim of the present study was to determine the agreement among four evaluators and compare the overall scores awarded when assessing portfolios of endodontic preclinical treatments performed by dental students by using an analytic rubric and a numeric rating scale.

Methods: A random sample of 42 portfolios performed by fourth-year dental students at preclinical endodontic practices were blindly assessed by four evaluators using two different evaluation methods: an analytic rubric specifically designed and a numeric rating scale. Six categories were analyzed: radiographic assessment, access preparation, shaping procedure, obturation, content of the portfolio, and presentation of the portfolio. The maximum global score was 10 points. The overall scores obtained with both methods from each evaluator were compared by Student's t, while agreement among evaluators was measured by Intraclass correlation coefficients (ICC). The influence of the difficulty of the endodontic treatment on the evaluators´ scores was analyzed by one-way ANOVA. Statistical tests were performed at a pre-set alpha of 0.05 using Stata 16.

Results: Difficulty of canal treatment did not influence the scores of evaluators, irrespective of the evaluation method used. When the analytic rubric was used, inter-evaluator agreement was substantial for radiographic assessment, access preparation, shaping procedure, obturation, and overall scores. Inter-evaluator agreement ranged from moderate to fair with the numeric rating scale. Mean higher overall scores were achieved when numeric rating scale was used. Presentation and content of the portfolio showed slight and fair agreement, respectively, among evaluators, regardless the evaluation method applied.

Conclusions: Assessment guided by an analytic rubric allowed evaluators to reach higher levels of agreement than those obtained when using a numeric rating scale. However, the rubric negatively affected overall scores.

Keywords: Dental education; Endodontics training; Evaluators´ agreement; Portfolio evaluation; Rubric.

Fig. 1

Distribution of fail-pass scores given by the evaluators (E1, E2, E3, E4) (n = 42) using a rubric (R) and a numeric rating scale (NRS)

Fig. 2

Distribution of qualifications (failed, approved, remarkable, outstanding) given by the evaluators (E1, E2, E3, E4) (n = 42) using a rubric (R) and a numeric rating scale (NRS)