STEM exam performance: Open- versus closed-book methods in the large language model era

Abstract

Background: The COVID-19 pandemic accelerated the shift to remote learning, heightening scrutiny of open-book examinations (OBEs) versus closed-book examinations (CBEs) within science, technology, engineering, arts and mathematics (STEM) education. This study evaluates the efficacy of OBEs compared to CBEs on student performance and perceptions within STEM subjects, considering the emerging influence of sophisticated large language models (LLMs) such as GPT-3.

Methods: Adhering to PRISMA guidelines, this systematic review analysed peer-reviewed articles published from 2013, focusing on the impact of OBEs and CBEs on university STEM students. Standardised mean differences were assessed using a random effects model, with heterogeneity evaluated by I² statistics, Cochrane's Q test and Tau statistics.

Results: Analysis of eight studies revealed mixed outcomes. Meta-analysis showed that OBEs generally resulted in better scores than CBEs, despite significant heterogeneity (I² = 97%). Observational studies displayed more pronounced effects, with noted concerns over technical difficulties and instances of cheating.

Discussion: Results suggest that OBEs assess competencies more aligned with current educational paradigms than CBEs. However, the emergence of LLMs poses new challenges to OBE validity by simplifying the generation of comprehensive answers, impacting academic integrity and examination fairness.

Conclusions: While OBEs are better suited to contemporary educational needs, the influence of LLMs on their effectiveness necessitates further study. Institutions should prudently consider the competencies assessed by OBEs, particularly in light of evolving technological landscapes. Future research should explore the integrity of OBEs in the presence of LLMs to ensure fair and effective student evaluations.

FIGURE 1

PRISMA flow chart. PRISMA flow diagram demonstrating the process for inclusion of selected papers in the review.

FIGURE 2

(a) Main meta‐analysis. Overall forest plot of mean exam scores in closed book settings versus open book settings shows a significant difference (p < 0.01). CBE, closed book exam; CI, closed book exam; IV, inverse variance weighted; MD, mean difference; OBE, open book exam. (b) Sub‐group meta‐analysis plot. Forest plot divided into sub‐groups based on study design. No significant sub‐group difference is noted by the Chi² test (p = 0.21).