Recognizing Reflection: Computer-Assisted Analysis of First Year Medical Students' Reflective Writing

Abstract

Background: Reflective writing is used throughout medical education to help students navigate their transformation into medical professionals. Assessment of reflective writing, however, is challenging; each available methodology of assessment has distinct advantages and disadvantages. We tested if combining two independent assessment mechanisms-a faculty-designed rubric and Academic Writing Analytics (AWA), an automated technique-could be used together to form a more robust form of evaluation.

Methods: We obtained reflective essays written by first year medical students as part of a clinical skills course. Faculty scored essays using a rubric designed to evaluate Integration, Depth, and Writing. The same essays were subjected to AWA analysis, which counted the number of reflective phrases indicative of Context, Challenge, or Change.

Results: Faculty scored the essays uniformly high, indicating that most students met the standard for reflection as described by the rubric. AWA identified over 1400 instances of reflective behavior within the essays, and there was significant variability in how often different types of reflective phrases were used by individual students.

Conclusions: While data from faculty assessment or AWA alone is sufficient to evaluate reflective essays, combining these methods offer a richer and more valuable understanding of the student's reflection.

Keywords: Natural language processing; Reflective writing; Undergraduate medical education.

Fig. 1

Numerical analysis of reflective essays. a Average score (in percent) per student (n = 106 students, 212 total essays) in each domain (Integration, Depth, and Writing) in the faculty rubric. The average score between Depth (95%) and Writing (97.5%) was statistically significantly different (*p < 0.05). No statistically significant difference was observed between Integration (average score = 96.3%) and Depth or Writing. b Average number of AWA counts (n = 106 students, 212 total essays) for each type of move. Students had 4.9 Context moves/student, 7.4 Challenge moves/student, and 1.0 Change moves/student. The difference between these averages was statistically significant (**p < 0.001). The top and bottom of the box are the standard deviation of the score. The caps on the vertical line are the minimum and maximum score. *p < 0.05; **p < 0.001. c Histogram of the percent of the class versus the average rubric score for each domain (Integration, Writing, and Depth). d Histogram of the percent of the class versus the number AWA counts for each domain (Context, Challenge, and Change). One hundred six students were included in the study. e Scores within each rubric domain for each individual student. f AWA counts of each type of reflective move for each individual student