Identification of parameters for electronic distance examinations

Abstract

Introduction: This study investigates the log data and response behavior from invigilated in-person electronic timed exams at the University of Veterinary Medicine Hannover, Foundation, Germany. The primary focus is on understanding how various factors influence the time needed per exam item, including item format, item difficulty, item discrimination and character count. The aim was to use these results to derive recommendations for designing timed online distance examinations, an examination format that has become increasingly important in recent years.

Methods: Data from 216,625 log entries of five electronic exams, taken by a total of 1,241 veterinary medicine students in 2021 and 2022, were analyzed. Various statistical methods were employed to assess the correlations between the recorded parameters.

Results: The analysis revealed that different item formats require varying amounts of time. For instance, image-based question formats and Kprim necessitated more than 60 s per item, whereas one-best-answer multiple-choice questions (MCQs) and individual Key Feature items were effectively completed in less than 60 s. Furthermore, there was a positive correlation between character count and response time, suggesting that longer items require more time. A negative correlation could be verified for the parameters "difficulty" and "discrimination index" towards response time, indicating that more challenging items and those that are less able to differentiate between high- and low-performing students take longer to answer.

Conclusion: The findings highlight the need for careful consideration of the ratio of item formats when defining time limits for exams. Regarding exam design, the literature mentions that time pressure is a critical factor, since it can negatively impact students' exam performance and some students, such as those with disabilities, are particularly disadvantaged. Therefore, this study emphasizes finding the right time limits to provide sufficient time for answering questions and reducing time pressure. In the context of unsupervised online exams, the findings of this study support previous recommendations that implementation of a stringent time limit might be a useful strategy to reduce cheating.

Keywords: E-assessment; examinations; item formats; log data; open-book; response time; veterinary education.

Figure 1

Absolute number of examinees submitting their exams at the same point in time for each of five anonymized academic subjects of veterinary medicine (A–E) including information on number of items and exam time limit. Submission times of exams by individual students were calculated as a percentage of the maximum available exam time. Examination difficulty is determined as the mean examination score of all examinees of the respective exam and shown as a percentage of the highest achievable score. Spearman’s rank correlation analysis shows a significant (p < 0.0001) negative relationship (r_s = −0.2738) between examination difficulty and submission time of the students.

Figure 2

Item response times of the five item formats MCQ, Kprim, Key Feature, picture diagnosis, and picture mapping. Kruskal-Wallis test displays significant differences (p < 0.0001) between the response times of item formats. Pairwise comparisons using the Dwass-Steel-Critchlow-Fligner method reveal statistically significant differences between MCQ and Kprim (p < 0.0001) as well as between Key Feature and Kprim (p < 0.0001); n = 346.

Figure 3

Item response times in minutes depending on the respective character count and separated by item format MCQ (n = 231, blue) and Kprim (n = 81, gray). Spearman’s rank correlation analysis indicates a statistically significant positive correlation between item response time and character count for MCQ (p < 0.0001, r_s = 0.3809) and Kprim (p < 0.0001, r_s = 0.5986); n = 312.

Figure 4

Item response times in minutes depending on the respective item difficulty index and separated by item format MCQ (n = 231, blue) and Kprim (n = 81, gray). Spearman’s rank correlation analysis indicates a statistically significant negative correlation between item response time and difficulty index for MCQ (p < 0.0001, r_s = −0.6607) and Kprim (p < 0.0001, r_s = −0.5038); n = 312.

Figure 5

Item response times in minutes depending on the respective item discrimination index and separated by item format MCQ (n = 231, blue) and Kprim (n = 81, gray). Spearman’s rank correlation analysis shows a statistically significant negative correlation between item response time and discrimination index for MCQ (p < 0.0001, r_s = −0.3779) and Kprim (p = 0.0099, r_s = −0.2851); n = 312.

Figure 6

Item discrimination indexes of MCQ items (n = 231) depending on their respective character count. Spearman’s rank correlation analysis displays a statistically significant negative correlation between discrimination index and character count (p = 0.0236, r_s = −0.1496); n = 231.