Can artificial intelligence write science? A comparative analysis of human-written and artificial intelligence-generated scientific writings

Abstract

Objective: Artificial intelligence (AI) is increasingly capable of academic writing, with large language models such as ChatGPT showing potential to assist or even generate scientific manuscripts. However, concerns remain regarding the quality, reliability, and interpretive capabilities of AI-generated content. The authors' study aimed to compare the quality of a human-written versus an AI-generated scientific manuscript to evaluate the strengths and limitations of AI in the context of academic publishing.

Methods: Two manuscripts were developed using identical titles, abstracts, and tables of a simulated analysis: one authored by a physician with multiple publications, and the other generated by ChatGPT-4o. Three independent and blinded reviewers-two human and one AI-assessed each manuscript across five domains: clarity and readability, coherence and flow, technical accuracy, depth, and conciseness and precision. Each category was scored on a 10-point scale, and qualitative feedback was collected to highlight specific strengths and weaknesses. Additionally, all reviewers were asked to deduce authorship of the manuscripts.

Results: The AI-generated manuscript scored higher in clarity and readability (mean 9.0 vs 7.2), but lower in technical accuracy (mean 6.3 vs 9.3) and depth (mean 5.5 vs 7.5). However, reviewers noted that the AI version lacked depth, critical analysis, and contextual interpretation. All reviewers accurately identified the authorship of each manuscript and tended to rate the version more favorably when it aligned with their own origin (human or AI); i.e., human reviewers assigned higher scores to the human-written manuscript, while the AI reviewer scored the AI-generated manuscript higher.

Conclusions: Although AI models can improve some aspects of scientific writing, particularly clarity and readability, they fall short in critical reasoning and contextual understanding. This reinforces the importance of human authorship and oversight in maintaining the critical analysis and scientific accuracy essential for academic publishing. AI may be used as a complementary tool to support, rather than replace, human-led scientific writing.

Keywords: ChatGPT; academic publishing; artificial intelligence; scientific writing; text generation; writing quality.