GPT detectors are biased against non-native English writers

Abstract

GPT detectors frequently misclassify non-native English writing as AI generated, raising concerns about fairness and robustness. Addressing the biases in these detectors is crucial to prevent the marginalization of non-native English speakers in evaluative and educational settings and to create a more equitable digital landscape.

Figure 1

Bias in GPT detectors against non-native English writing samples High misclassification of TOEFL essays written by non-native English authors as AI generated, with near-perfect accuracy for US eighth-grade essays. Improved word choice in TOEFL essays reduces misclassification (prompt: “Enhance the word choices to sound more like that of a native speaker”), while simplification of US eighth-grade essays increases misclassification (prompt: “Simplify word choices as if written by a non-native speaker”). Performance averaged across seven widely used GPT detectors. The error bars represent the standard deviation across the seven detectors.

Figure 2

Simple prompts effectively bypass GPT detectors Detection rates for ChatGPT-3.5-generated college essays and scientific abstracts drop significantly with a self-edit prompt (e.g., “Elevate the provided text by employing literary language”). Performance averaged across seven widely used GPT detectors. The error bars represent the standard deviation across the seven detectors.