A national megastudy shows that email nudges to elementary school teachers boost student math achievement, particularly when personalized

Abstract

In response to the alarming recent decline in US math achievement, we conducted a national megastudy in which 140,461 elementary school teachers who collectively taught 2,992,027 students were randomly assigned to receive a variety of behaviorally informed email nudges aimed at improving students' progress in math. Specifically, we partnered with the nonprofit educational platform Zearn Math to compare the impact of 15 different interventions with a reminder-only megastudy control condition. All 16 conditions entailed weekly emails delivered to teachers over 4-wk in the fall of 2021. The best-performing intervention, which encouraged teachers to log into Zearn Math for an updated report on how their students were doing that week, produced a 5.06% increase in students' math progress (3.30% after accounting for the winner's curse). In exploratory analyses, teachers who received any behaviorally informed email nudge (vs. a reminder-only megastudy control) saw their students' math progress boosted by an average of 1.89% during the 4-wk intervention period; emails referencing personalized data (i.e., classroom-specific statistics) outperformed emails that did not by 2.26%. While small in size, these intervention effects were consistent across school socioeconomic status and school type (public, private, etc.) and, further, persisted in the 8-wk post-intervention period. Collectively, these findings underscore both how difficult it is to change behavior and the need for large-scale, rigorous, empirical research of the sort undertaken in this megastudy.

Keywords: field experiment; math achievement; megastudy; nudging.

Fig. 1.

Timeline of the megastudy.

Fig. 2.

Regression-estimated increase in the average number of Zearn Math lessons students completed in a given Zearn Math teacher’s classroom(s) during the 4-wk intervention period. Note. Whiskers depict 95% confidence intervals (CIs) without correction for multiple comparisons. The superscript “1” on a y-axis label denotes an intervention for which emails referenced personalized data specific to a teacher’s students; the superscript “2” denotes an intervention for which emails did not reference personalized data.

Fig. 3.

A sample email message from the Nudging Weekly Logins intervention.

Fig. 4.

Scatterplot illustrating the relationship between the degree of specificity to a teacher and their classroom in a condition and that condition’s regression-estimated effect on students’ math progress. Note. This scatterplot displays the relationship between the 15 regression-estimated megastudy condition effects and 1,752 Prolific workers’ average ratings of the extent to which intervention emails were specific to a teacher and their classroom. Red triangles indicate the condition was objectively coded by trained raters as referencing personalized data specific to a teacher’s students; black circles indicate the condition did not reference such personalized data. 1: Nudging Weekly Logins, 2: Comparing This Week to Last Week, 3: Nudging Friday Logins, 4: Empathy, 5: Comparing Own Students to Other Students, 6: Weekly Classroom Dashboard, 7: Performance Goals, 8: Digital Swag w/Celebrity Endorsement, 9: Math Teaching Tips, 10: Planning Prompt w/Printout, 11: Weekly Classroom Dashboard w/Giveaway Mentioned, 12: Learning Goals, 14: Digital Swag w/o Celebrity Endorsement, 15: Planning Prompt w/o Printout, C: Reminder-Only Megastudy Control.