How AI should be used in radiology: assessing ambiguity and completeness of intended use statements of commercial AI products

Abstract

Background: Intended use statements (IUSs) are mandatory to obtain regulatory clearance for artificial intelligence (AI)-based medical devices in the European Union. In order to guide the safe use of AI-based medical devices, IUSs need to contain comprehensive and understandable information. This study analyzes the IUSs of CE-marked AI products listed on AIforRadiology.com for ambiguity and completeness.

Methods: We retrieved 157 IUSs of CE-marked AI products listed on AIforRadiology.com in September 2022. Duplicate products (n = 1), discontinued products (n = 3), and duplicate statements (n = 14) were excluded. The resulting IUSs were assessed for the presence of 6 items: medical indication, part of the body, patient population, user profile, use environment, and operating principle. Disclaimers, defined as contra-indications or warnings in the IUS, were identified and compared with claims.

Results: Of 139 AI products, the majority (n = 78) of IUSs mentioned 3 or less items. IUSs of only 7 products mentioned all 6 items. The intended body part (n = 115) and the operating principle (n = 116) were the most frequently mentioned components, while the intended use environment (n = 24) and intended patient population (n = 29) were mentioned less frequently. Fifty-six statements contained disclaimers that conflicted with the claims in 13 cases.

Conclusion: The majority of IUSs of CE-marked AI-based medical devices lack substantial information and, in few cases, contradict the claims of the product.

Critical relevance statement: To ensure correct usage and to avoid off-label use or foreseeable misuse of AI-based medical devices in radiology, manufacturers are encouraged to provide more comprehensive and less ambiguous intended use statements.

Key points: • Radiologists must know AI products' intended use to avoid off-label use or misuse. • Ninety-five percent (n = 132/139) of the intended use statements analyzed were incomplete. • Nine percent (n = 13) of the intended use statements held disclaimers contradicting the claim of the AI product. • Manufacturers and regulatory bodies must ensure that intended use statements are comprehensive.

Keywords: Artificial intelligence; Device approval process; Medical device legislation; Medical device safety; Radiology.

Fig. 1

Histogram of the number of items present per intended use statement ranging from 0 to 6 items. Intended use items scored were intended medical indication, intended part of the body or type of tissue applied to or interacted with, intended patient population, intended user profile, intended use environment, and intended operating principle

Fig. 2

Two spider charts of the percentage of IUS items present per regulatory class of the product (a) and product type (b). MDD, Medical Devices Directive; MDR, Medical Devices Regulation

Fig. 3

Bar chart of the number of intended use statements without disclaimers (blue), with disclaimers that did not interfere with the claim of the product (orange), and disclaimers that contradict the claim of the product (gray), per regulatory class and product type. MDD, Medical Devices Directive; MDR, Medical Devices Regulation