Accuracy of Artificial Intelligence vs Professionally Translated Discharge Instructions

Abstract

Importance: Patients using languages other than English are a group at risk of poor health outcomes and encounter barriers to access of translated written materials. Although artificial intelligence (AI) may offer an opportunity to improve access, few studies have evaluated the accuracy and safety of AI translation for clinical care under routine practice conditions.

Objective: To investigate the accuracy of AI translation compared with professional human translation of patient-specific issued pediatric inpatient discharge instructions.

Design, setting, and participants: This comparative effectiveness analysis compared translations by a neural machine translation model vs professional translators using patient-specific pediatric inpatient discharge instructions received by families between May 18, 2023, and May 18, 2024, at a single center academic pediatric hospital. Instructions were translated to Simplified Chinese, Somali, Spanish, and Vietnamese by professional translators and the Azure AI system and then broken into scoring sections. Two professional translators per language evaluated translations (blinded to source) on an established 5-point scale for fluency, adequacy, meaning, and error severity, with 1 indicating worst performance and 5 indicating best performance.

Exposure: AI vs professional translation.

Main outcome and measure: Quality of discharge instruction translation, including fluency, adequacy, meaning, and severity of errors.

Results: A total of 148 sections from 34 discharge instructions were analyzed. When considering all 4 languages together, average fluency, adequacy, and meaning were lower among AI compared with professional human translations. Among all tested languages, mean (SD) fluency for AI translations was 2.98 (1.12) compared with 3.90 (0.96) for professional translations (difference, 0.92; 95% CI, 0.83-1.01; P < .001), adequacy was 3.81 (1.14) compared with 4.56 (0.70) (difference, 0.74; 95% CI, 0.65-0.83; P < .001), meaning was 3.38 (1.15) compared with 4.28 (0.84) (difference, 0.90; 95% CI, 0.80-0.99; P < .001), and error severity was 3.53 (1.28) compared with 4.48 (0.88) (difference, 0.95; 95% CI, 0.85-1.06; P < .001). Compared with professional translations, the Spanish AI translations were noninferior in adequacy (difference, 0.08; 95% CI, -0.02 to 0.19) and error severity (difference, 0.03; 95% CI, -0.09 to 0.14) but inferior in fluency (difference, 0.38; 95% CI, 0.23-0.53) and just crossed the inferiority threshold in meaning (difference, 0.08; 95% CI, -0.04 to 0.20). The Chinese, Vietnamese, and Somali AI translations were inferior to the professional translations across all metrics, with the greatest differences for Somali.

Conclusions and relevance: In this comparative effectiveness analysis of AI- vs professionally translated issued discharge instructions, AI-translated instructions performed similarly for Spanish but worse for other languages tested. Validation and clinical implementation of AI-based translation will require special attention to languages of lesser diffusion to prevent creating new inequities.

Figure 1.. Comparison of Mean Fluency, Adequacy, Meaning, and Error Severity Scores for Professionally Translated vs Artificial Intelligence (AI)–Translated Discharge Instructions

Discharge instructions (n = 148 for each category) were scored by professional translators blinded to source in 4 languages. Scores were assigned on a scale of 1 to 5, with 1 indicating worst performance and 5 indicating best performance. P values are based on paired t tests testing for differences between professional and AI translation of issued discharge instructions (rather than noninferiority). Significant differences were found between professional and AI translations in all categories for all languages other than Spanish, where the only significant difference was found in fluency.

Figure 2.. Frequency of Clinically Impactful Errors by Language and Translation Type

Discharge instructions were scored by professional translators blinded to source in 4 languages (n = 148 per category). Scores were assigned on a scale of 1 to 5, with 1 indicating most harmful (harms patient) and 4 to 5 indicating not clinically impactful. P values are based on paired Wilcoxon signed-rank testing for significant differences between professional and artificial intelligence (AI) translation of issued discharge instructions (rather than noninferiority). No significant differences were found in proportions of errors in Spanish, but all other groups showed significantly worse performance of the AI.