Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Nov 21;40(1):e61.
doi: 10.1017/S0266462324000308.

Health technology assessment framework for artificial intelligence-based technologies

Rossella Di Bidino  1   2 , Signe Daugbjerg  1 , Sara C Papavero  1 , Ira H Haraldsen  3 , Americo Cicchetti  4 , Dario Sacchini  5   6
Affiliations

Health technology assessment framework for artificial intelligence-based technologies

Rossella Di Bidino et al. Int J Technol Assess Health Care. .

Abstract

Objectives: Artificial intelligence (AI)-based health technologies (AIHTs) have already been applied in clinical practice. However, there is currently no standardized framework for evaluating them based on the principles of health technology assessment (HTA).

Methods: A two-round Delphi survey was distributed to a panel of experts to determine the significance of incorporating topics outlined in the EUnetHTA Core Model and twenty additional ones identified through literature reviews. Each panelist assigned scores to each topic. Topics were categorized as critical to include (scores 7-9), important but not critical (scores 4-6), and not important (scores 1-3). A 70 percent cutoff was used to determine high agreement.

Results: Our panel of 46 experts indicated that 48 out of the 65 proposed topics are critical and should be included in an HTA framework for AIHTs. Among the ten most crucial topics, the following emerged: accuracy of the AI model (97.78 percent), patient safety (95.65 percent), benefit-harm balance evaluated from an ethical standpoint (95.56 percent), and bias in data (91.30 percent). Importantly, our findings highlight that the Core Model is insufficient in capturing all relevant topics for AI-based technologies, as 14 out of the additional 20 topics were identified as crucial.

Conclusion: It is imperative to determine the level of agreement on AI-relevant HTA topics to establish a robust assessment framework. This framework will play a foundational role in evaluating AI tools for the early diagnosis of dementia, which is the focus of the European project AI-Mind currently being developed.

Keywords: AI-HTA framework; AI-Mind Study; artificial intelligence; health technology assessment; value assessment.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1.
Figure 1.
European Union countries represented in the panel.
Figure 2.
Figure 2.
Summary of results for each traditional domain according to the EUnetHTA Core Model.
Figure 3.
Figure 3.
Summary of results for additional topics.
Figure 4.
Figure 4.
Hierarchy of health technology assessment domains as perceived by panel of experts.
See this image and copyright information in PMC

References

    1. Ma B, Yang J, Wong FKY, et al. Artificial intelligence in elderly healthcare: A scoping review. Ageing Res Rev. 2023;83:101808. - PubMed
    1. Ciecierski-Holmes T, Singh R, Axt M, Brenner S, Barteit S. Artificial intelligence for strengthening healthcare systems in low- and middle-income countries: A systematic scoping review. NPJ Digit Med. 2022;28(1):162. - PMC - PubMed
    1. Organisation for Economic Co-operation and Development (OECD). [Internet] Updates to the OECD’s definition of an AI system explained [cited 2023 Dec 20]. 2023. Available from: https://oecd.ai/en/wonk/ai-system-definition-update.
    1. Organisation for Economic Co-operation and Development (OECD). [Internet] Recommendation of the Council on Artificial Intelligence [cited 2023 Dec 20]. 2019. Available from: https://oecd.ai/en/assets/files/OECD-LEGAL-0449-en.pdf.
    1. Estevez Almenzar M, Fernandez Llorca D, Gomez Gutierrez E, Martinez Plumed F. Glossary of human-centric artificial intelligence. Luxembourg: Publications Office of the European Union; 2022.

MeSH terms

LinkOut - more resources