A model-independent redundancy measure for human versus ChatGPT authorship discrimination using a Bayesian probabilistic approach

Abstract

The academic and scientific world in general is increasingly concerned about their inability to determine and ascertain the identity of the writer of a text. More and more often the question arises as to whether a scientific article or work handed in by a student was actually produced by the alleged author of the questioned text. The role of artificial intelligence (AI) is increasingly debated due to its dangers of undeclared use. A current example is undoubtedly the undeclared use of ChatGPT to write a scientific text. The article promotes an AI model-independent redundancy measure to support discrimination between hypotheses on authorship of various multilingual texts written by humans or produced by intelligence media such as ChatGPT. The syntax of texts written by humans tends to differ from that of texts produced by AIs. This difference can be grasped and quantified even with short texts (i.e. 1800 characters). This aspect of length is extremely important, because short texts imply a greater difficulty of analysis to characterize authorship. To meet the efficiency criteria required for the evaluation of forensic evidence, a probabilistic approach is implemented. In particular, to assess the value of the redundancy measure and to offer a consistent classification criterion, a metric called Bayes factor is implemented. The proposed Bayesian probabilistic method represents an original approach in stylometry. Analyses performed over multilingual texts (English and French) covering different scientific and human areas of interest (forensic science and socio-psycho-artistic topics) reveal the feasibility of a successful authorship discrimination with limited misclassification rates. Model performance is satisfactory even with small sample sizes.

Figure 1

Redundancy measure for words uni- (U), bi- (B), tri- (T) and quadri- (Q) grams in the two populations: forensic science papers (FSI, left) and student’s manuscripts (Students, right). A distinction is made for texts written by humans (h), either forensic scientists or students (blue colored boxplots), and text delivered by the artificial intelligence (c), either for scientific papers or students’ texts (red colored boxplots). The first population (FSI) is characterized by texts written in English, while the second one (Students) is characterized by texts written in French.

Figure 2

Weights of evidence, log(BF), for every texts written by (1) ChatGPT on forensic (solid red-colored line) and socio-psycho-artistic themes (dashed red-colored line) and by (2) humans scientists (solid blue-colored line) and students (dashed blue-colored line).