AI-driven approach for creating and evaluating a synthetic dataset for Medication Errors

Abstract

Objective: This study aims to create a complete Medication Error (ME) dataset. This will help to address the challenge of limited access to real-world data for developing machine learning models in healthcare applications.

Methods: We use transformer-based models (GPT-4, LLAMA3, and Mistral) to create our synthetic dataset in French. These models generate a diverse range of descriptions that capture the variability of ME types. We assess the effectiveness of our synthetic dataset through expert evaluations by healthcare professionals and an AI-driven analysis, to test its realism and its utility in training machine learning models for ME classification.

Results: The synthetic dataset demonstrates high accuracy and realism in representing diverse ME scenarios. Expert evaluation confirms that the dataset is similar to real-world ME data. The AI-driven evaluation also shows that models trained on synthetic data achieved robust classification performance, validating the dataset's utility for the development of effective ME classification tools.

Conclusion: The proposed approach demonstrates the potential of large language models to generate realistic synthetic ME reports in French. Out of 200 evaluated reports, 70% of zero-shot outputs were deemed below expectations, while 80% of one-shot and few-shot outputs were considered valid or valid with minor revisions by clinical experts. Furthermore, classifiers trained on 800 synthetic reports attained an F1-score of up to 0.78 when tested on real data. These results confirm that synthetic data can effectively support AI-driven ME analysis in contexts where real-world data is limited or unavailable.

Keywords: Generative AI; Medication Error; Synthetic data; Transformer model.