Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2025 Feb;22(1):e13839.
doi: 10.1111/tct.13839. Epub 2024 Nov 4.

STEM exam performance: Open- versus closed-book methods in the large language model era

Rasi Mizori  1 , Muhayman Sadiq  1 , Malik Takreem Ahmad  1 , Anthony Siu  1 , Reubeen Rashid Ahmad  2 , Zijing Yang  1 , Helen Oram  1 , James Galloway  1
Affiliations
Comparative Study

STEM exam performance: Open- versus closed-book methods in the large language model era

Rasi Mizori et al. Clin Teach. 2025 Feb.

Abstract

Background: The COVID-19 pandemic accelerated the shift to remote learning, heightening scrutiny of open-book examinations (OBEs) versus closed-book examinations (CBEs) within science, technology, engineering, arts and mathematics (STEM) education. This study evaluates the efficacy of OBEs compared to CBEs on student performance and perceptions within STEM subjects, considering the emerging influence of sophisticated large language models (LLMs) such as GPT-3.

Methods: Adhering to PRISMA guidelines, this systematic review analysed peer-reviewed articles published from 2013, focusing on the impact of OBEs and CBEs on university STEM students. Standardised mean differences were assessed using a random effects model, with heterogeneity evaluated by I2 statistics, Cochrane's Q test and Tau statistics.

Results: Analysis of eight studies revealed mixed outcomes. Meta-analysis showed that OBEs generally resulted in better scores than CBEs, despite significant heterogeneity (I2 = 97%). Observational studies displayed more pronounced effects, with noted concerns over technical difficulties and instances of cheating.

Discussion: Results suggest that OBEs assess competencies more aligned with current educational paradigms than CBEs. However, the emergence of LLMs poses new challenges to OBE validity by simplifying the generation of comprehensive answers, impacting academic integrity and examination fairness.

Conclusions: While OBEs are better suited to contemporary educational needs, the influence of LLMs on their effectiveness necessitates further study. Institutions should prudently consider the competencies assessed by OBEs, particularly in light of evolving technological landscapes. Future research should explore the integrity of OBEs in the presence of LLMs to ensure fair and effective student evaluations.

PubMed Disclaimer

Conflict of interest statement

The authors declare no potential conflicts of interest with respect to the publication, research, and/or authorship of this article.

Figures

FIGURE 1
FIGURE 1
PRISMA flow chart. PRISMA flow diagram demonstrating the process for inclusion of selected papers in the review.
FIGURE 2
FIGURE 2
(a) Main meta‐analysis. Overall forest plot of mean exam scores in closed book settings versus open book settings shows a significant difference (p < 0.01). CBE, closed book exam; CI, closed book exam; IV, inverse variance weighted; MD, mean difference; OBE, open book exam. (b) Sub‐group meta‐analysis plot. Forest plot divided into sub‐groups based on study design. No significant sub‐group difference is noted by the Chi2 test (p = 0.21).
See this image and copyright information in PMC

References

    1. Theophilides C, Koutselini M. Study behavior in the closed‐book and the open‐book examination: a comparative analysis. Educ Res Eval. 2000;21(1):379–393. Available at: 10.1076/EDRE.6.4.379.6932 - DOI
    1. Krasne S, Wimmers PF, Relan A, Drake TA. Differential effects of two types of formative assessment in predicting performance of first‐year medical students. Adv Health Sci Educ. 2006;11(2):155–171. 10.1007/S10459-005-5290-9/METRICS - DOI - PubMed
    1. Agarwal PK, Karpicke JD, Kang SHK, Roediger HL III, McDermott KB. Examining the testing effect with open‐ and closed‐book tests. Appl Cogn Psychol. 2008;22(7):861–876. 10.1002/ACP.1391 - DOI
    1. Dave M, Patel K, Patel N. A systematic review to compare open and closed book examinations in medicine and dentistry. Fac Dent J. 2021;12(4):174–180. 10.1308/rcsfdj.2021.41 - DOI
    1. Durning SJ, Dong T, Ratcliffe T, Schuwirth L, Artino AR Jr, Boulet JR, et al. Comparing open‐book and closed‐book examinations: a systematic review. Acad Med. 2016;91(4):583–599. 10.1097/ACM.0000000000000977 - DOI - PubMed

LinkOut - more resources