Scorecard for synthetic medical data evaluation

Abstract

Although the interest in synthetic medical data (SMD) for developing and testing artificial intelligence (AI) methods is growing, the absence of a comprehensive framework to evaluate the quality and applicability of SMD hinders its wider adoption. Here, we outline an evaluation framework designed to meet the unique requirements of medical applications. We also introduce SMD scorecard, a comprehensive report accompanying artificially generated datasets. This scorecard provides a quantitative assessment of SMD across seven criteria (7 Cs), complemented by a descriptive section that contains all relevant information about the dataset. The SMD scorecard provides a practical framework for evaluating and reporting the quality of synthetic data, which can benefit SMD developers and users.

Fig. 1. Evaluation challenges in synthetic medical images and textual data.

a A synthetic image produced by a generative AI model (adopted from Ref. with high visual fidelity but containing structural inconsistencies such as broken ligaments, b, c Synthetic chest X-rays ( featuring misplaced medical devices (pacemakers and tubes) located outside of anatomically plausible regions, and (c) digital mammograms that score highly on fidelity and statistical metrics but display clinically implausible artifacts including abnormal breast shape and the erroneous presence of multiple nipple-like structures indicated by the red arrows. (d) shows textual outputs from LLMs in a medical query task where the LLM provides incorrect responses to critical clinical questions which can often be detected through direct comparison with ground truth. However, in the summarization task (e), the LLM generates a symptom (leg swelling) not present in the original EHR note, which is harder to detect using similarity- or overlap-based metrics.

Fig. 2. Scorecard for evaluating and reporting synthetic medical data.

Congruence measures the alignment between the distributions of synthetic and real data. Coverage highlights the variability in synthetic data, demonstrated using convex hull volume, where synthetic data shows a smaller spread compared to patient data. Constraint evaluates adherence to clinical context (e.g., tumor size and density) by identifying valid versus invalid data points. Completeness assesses the presence of necessary information for the intended use. Compliance ensures adherence to local and global standards such as de-identification, privacy preservation, and file format consistency. Comprehension evaluates the accessibility and clarity of SMD generation processes and documentation. The bottom panel illustrates the consistency across subgroups (e.g., demographic or disease-specific groups). Essential descriptive information about the synthetic data is presented in the middle panel.