Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Oct 2;21(1):274.
doi: 10.1186/s12911-021-01634-3.

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Rosy Tsopra  1   2   3   4 , Xose Fernandez  5 , Claudio Luchinat  6 , Lilia Alberghina  7 , Hans Lehrach  8   9 , Marco Vanoni  7 , Felix Dreher  9 , O Ugur Sezerman  10 , Marc Cuggia  11 , Marie de Tayrac  12 , Edvins Miklasevics  13 , Lucian Mihai Itu  14 , Marius Geanta  15 , Lesley Ogilvie  8   9 , Florence Godey  16   17 , Cristian Nicolae Boldisor  14 , Boris Campillo-Gimenez  18 , Cosmina Cioroboiu  15 , Costin Florian Ciusdel  14 , Simona Coman  14 , Oliver Hijano Cubelos  5 , Alina Itu  14 , Bodo Lange  9 , Matthieu Le Gallo  16   17 , Alexandra Lespagnol  19 , Giancarlo Mauri  20 , H Okan Soykam  21 , Bastien Rance  22   23   24 , Paola Turano  6 , Leonardo Tenori  6 , Alessia Vignoli  6 , Christoph Wierling  9 , Nora Benhabiles  25 , Anita Burgun  22   23   24   26
Affiliations

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Rosy Tsopra et al. BMC Med Inform Decis Mak. .

Abstract

Background: Artificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.

Methods: The European "ITFoC (Information Technology for the Future Of Cancer)" consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.

Results: This framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the "ITFoC Challenge". This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.

Conclusions: The predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

Keywords: Artificial intelligence; Cancer; Computerized decision support systems; Oncology; Personalized medicine; Precision medicine.

PubMed Disclaimer

Conflict of interest statement

Hans Lehrach is a member of the board of Alacris Theranostics GmbH. Felix Dreher is an employee of Alacris Theranostics GmbH. Lesley Ogilvie is an employee of Alacris Theranostics GmbH. Bodo Lange is the CEO of Alacris Theranostics GmbH. Christoph Wierling is an employee of Alacris Theranostics GmbH. The other authors have no conflicts of interest to declare.

Figures

Fig. 1
Fig. 1
The seven key steps needed for the clinical validation of AI technologies
Fig. 2
Fig. 2
Evaluation of AI-timing
Fig. 3
Fig. 3
Data workflow for the ITFoC challenge
See this image and copyright information in PMC

References

    1. Paton C, Kobayashi S. An open science approach to artificial intelligence in healthcare. Yearb Med Inform. 2019;28:47–51. - PMC - PubMed
    1. Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6:94–98. - PMC - PubMed
    1. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–2410. - PubMed
    1. Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018;1:39. - PMC - PubMed
    1. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–118. - PMC - PubMed

Publication types