Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning

Abstract

Clinical text and documents contain very rich information and knowledge in healthcare, and their processing using state-of-the-art language technology becomes very important for building intelligent systems for supporting healthcare and social good. This processing includes creating language understanding models and translating resources into other natural languages to share domain-specific cross-lingual knowledge. In this work, we conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three sub-tasks including (1) clinical case (CC), (2) clinical terminology (CT), and (3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) outperformed the other two extra-large language models by a large margin in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data is openly available for research purposes at: https://github.com/HECTA-UoM/ClinicalNMT.

Keywords: Neural machine translation; Spanish-English translation; clinical knowledge transformation; clinical text translation; large language model; multilingual pre-trained language model; transfer learning.

Figure 1

Illustration of the Investigation Workflow.

Figure 2

Marian Pre-Trained NMT - Training Pipeline.

Figure 3

Several Attention based Transformer NMT structure (16).

Figure 4

original dense Transformer (left) vs MoE Transformer (right) (13).

Figure 5

MoE vs Conditional MoE (13).

Figure 6

Comparison of Automatic Evaluations against Human Evaluation (HOPE).

Figure 7

Summary of Human Expert-Based Evaluations.

Figure 8

Confidence Intervals of Three Models (M, N, W): Clinical-Marian, Clinical-NLLB, and Clinical-WMT21fb.

Figure 9

Task-1 Cases/Sentences EN-ES Translation Examples: clinic-WMT21fb vs clinic-NLLB.

Figure 10

Task-2 Clinical Term ES-EN Translation Examples: clinic-WMT21fb vs clinic-NLLB.

Figure 11

Task-3 Concept EN-ES Translation Examples: clinic-WMT21fb vs clinic-NLLB.

Figure A1

MarianNMT Fine-Tuning Parameters: Encoder and Decoder with $6+6$ Layers.

Figure A2

M2M-100 Model Structure For Conditional Generation: Encoder and Decoder Parameters with $24+24$ Layers.