Complementing but Not Replacing: Comparing the Impacts of GPT-4 and Native-Speaker Interaction on Chinese L2 Writing Outcomes

Abstract

This study explored the efficacy of large language models (LLMs), namely GPT-4, in supporting second language (L2) writing in comparison with interaction with a human language partner in the pre-writing phase. A within-subject behavioral experiment was conducted with 23 Chinese L2 learners who were exposed to three conditions: "without interaction", "interaction with GPT-4", and "interaction with a language partner". They then completed an L2 writing task. It was found that interaction with the language partner yielded significantly improved results compared with both interaction with GPT-4 and the case without interaction in terms of overall writing scores, organization, and language. Additionally, both types of interaction enhanced the participants' topic familiarity and writing confidence and reduced the task's perceived difficulty compared with the case without interaction. Interestingly, in the "interaction with GPT-4" condition, topic familiarity was positively correlated with better writing outcomes, whereas in the "interaction with a language partner" condition, perceived difficulty was positively correlated with content scores; however, content scores were negatively associated with writing confidence. This study suggests that LLMs should be used to complement and not replace human language partners in the L2 pre-writing phase.

Keywords: GPT-4; human language partner; large language models; second language writing.

Figure 1

The experimental procedure. Each participant was invited to participate in a three-session writing experiment. Between each session, there was a minimum 10-day interval. In order to avoid carryover effects from the interactions, all participants first underwent a session involving writing without interaction; subsequently, the two interaction conditions (i.e., interaction with GPT-4 and interaction with the language partner) were counterbalanced across the participants. The writing topics were also counterbalanced across the three writing sessions.

Figure 2

Writing score comparison results. p-values for the Friedman test with effect sizes (W denotes Kendall’s W) and p-values for the Wilcoxon signed-rank test are provided. On the horizontal axis, W—without interaction; G—interaction with GPT-4; and P—interaction with language partner.

Figure 3

Rating score comparison results. p-values for the Friedman test with effect sizes (W denotes Kendall’s W) and p-values for the Wilcoxon signed-rank test are provided.

Figure 4

Spearman’s correlation test results. Spearman’s R values, as well as p-values, are provided. Post_W: rating score change, i.e., post-interaction rating score−rating score under the “without interaction” condition; G_W: writing score change, i.e., G = writing score under interaction with GPT-4−writing score under the “without interaction” condition; and P_W: writing score change, i.e., P = writing score under interaction with a language partner−writing score under the “without interaction” condition. (A) Heat map of the correlation between interaction with GPT-4 ratings (familiarity, confidence, and difficulty) and writing scores (total & sub-dimensions). (B) Heat map of the correlation between interaction with language partner ratings (familiarity, confidence, and difficulty) and writing scores (total & sub-dimensions). (C) Significant correlation between Gpost_W familiarity and writing scores (total & sub-dimensions). (D) Significant correlation between Ppost_W difficulty/confidence and content score.