Synthetic4Health: generating annotated synthetic clinical letters

Abstract

Clinical letters contain sensitive information, limiting their use in model training, medical research, and education. This study aims to generate reliable, diverse, and de-identified synthetic clinical letters to support these tasks. We investigated multiple pre-trained language models for text masking and generation, focusing on Bio_ClinicalBERT, and applied different masking strategies. Evaluation included qualitative and quantitative assessments, downstream named entity recognition (NER) tasks, and clinically focused evaluations using BioGPT and GPT-3.5-turbo. The experiments show: (1) encoder-only models perform better than encoder-decoder models; (2) models trained on general corpora perform comparably to clinical-domain models if clinical entities are preserved; (3) preserving clinical entities and document structure aligns with the task objectives; (4) Masking strategies have a noticeable impact on the quality of synthetic clinical letters: masking stopwords has a positive impact, while masking nouns or verbs has a negative effect; (5) The BERTScore should be the primary quantitative evaluation metric, with other metrics serving as supplementary references; (6) Contextual information has only a limited effect on the models' understanding, suggesting that synthetic letters can effectively substitute real ones in downstream NER tasks; (7) Although the model occasionally generates hallucinated content, it appears to have little effect on overall clinical performance. Unlike previous research, which primarily focuses on reconstructing original letters by training language models, this paper provides a foundational framework for generating diverse, de-identified clinical letters. It offers a direction for utilizing the model to process real-world clinical letters, thereby helping to expand datasets in the clinical domain. Our codes and trained models are available at https://github.com/HECTA-UoM/Synthetic4Health.

Keywords: clinical NLP (natural language processing); encoder-only models; encoder–decoder models; masking and generating; named entity recognation; pre-trained language models (PLMs); synthetic data creation.

Figure 1

An example of the objective: sentence/segment-level generations.

Figure 2

Overall investigation workflow for Synthetic4Health.

Figure 3

Pre-processing pipeline.

Figure 4

Text chunking workflow.

Figure 5

Comparison of encoder-only and encoder–decoder model architectures.

Figure 6

Evaluation pipeline.

Figure 7

Workflow of the downstream NER task.

Figure 8

GPT-3.5-turbo prompt for clinical evaluation.

Figure 9

Original unprocessed example sentence (–8) (“note_id”: “10807423-DS-19”) (the circled tokens will be masked).

Figure 10

An example of the masked sentence.

Figure 11

Example sentence generated by Bio_ClinicalBERT.

Figure 12

Example sentence generated by T5-base.

Figure 13

Example sentence 1 with different masked tokens.

Figure 14

Post-processing results with BERT-Base.