External validation of a fully automated evaluation tool: a retrospective analysis of 68,471 scored embryos

Abstract

Objective: To externally validate a fully automated embryo classification system for in vitro fertilization (IVF) treatments.

Design: Retrospective cohort study.

Subjects: A total of 6,434 patients undergoing 7,352 IVF treatments contributed 70,456 embryos.

Exposure: Embryos were evaluated by conventional morphology and retrospectively scored using a fully automated deep learning-based algorithm across conventional IVF, oocyte donation, and preimplantation genetic testing for aneuploidy (PGT-A) cycles.

Main outcome measures: The primary outcomes were implantation and live birth, including odds ratios (ORs) from generalized estimating equation models. Secondary outcomes were embryo morphology, euploidy, and miscarriage. Exploratory outcomes included a comparison between conventional methodology and artificial intelligence algorithm with areas under the receiver operating characteristics curves (AUCs), agreement degree between artificial intelligence and embryologists, Cohen's Kappa coefficient, and relative risk.

Results: Implantation and live birth rates increased as the automatic embryo scores increased. The generalized estimating equation model, controlling for confounders, showed that the automatic score was associated with an OR of 1.31 (95% confidence interval [CI], 1.25-1.36) for implantation in treatments using oocytes from patients and an OR of 1.17 (95% CI, 1.14-1.20) in the oocyte donation program, with no significant association with PGT-A treatments. For live birth, the ORs were 1.27 (95% CI, 1.21-1.33) for patients, 1.16 (95% CI, 1.13-1.19) for donors, and 1.05 (95% CI, 1-1.10) for PGT-A cycles. The average score was higher in embryos with better morphology, in euploid embryos compared with aneuploid embryos, and in embryos that resulted in a full-term pregnancy compared with those that miscarried. Concordance between the highest-scoring embryo and the embryo with the best conventional morphology was 71.4% (95% CI, 67.7%-75.0%) in treatments with patient oocytes and 61.0% (95% CI, 58.6%-63.4%) in the oocyte donation program. Overall, the Cohen's Kappa coefficient was 0.63. The automatic embryo score showed similar AUCs to conventional morphology, although implantation was higher when the transferred embryo matched the highest-scoring embryo from each cohort (57.36% vs. 49.98%). Relative risk indicated a 1.14-fold increase in implantation likelihood when the top-ranked embryo was transferred.

Conclusions: A fully automated embryo scoring system effectively ranked embryos based on their potential for implantation and live birth. The performance of the conventional methodology was comparable to that of the artificial intelligence-based technology; however, better clinical outcomes were observed when the highest-scoring embryo in the cohort was transferred.

Keywords: Automation; concordance; embryo selection; implantation; live birth.