Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use

Abstract

Importance: The primary objective of any newly developed medical device using artificial intelligence (AI) is to ensure its safe and effective use in broader clinical practice.

Objective: To evaluate key characteristics of AI-enabled medical devices approved by the US Food and Drug Administration (FDA) that are relevant to their clinical generalizability and are reported in the public domain.

Design, setting, and participants: This cross-sectional study collected information on all AI-enabled medical devices that received FDA approval and were listed on the FDA website as of August 31, 2024.

Main outcomes and measures: For each AI-enabled medical device, detailed information and key characteristics relevant for the generalizability of the devices at the time of approval were summarized, specifically examining clinical evaluation aspects, such as the presence and design of clinical performance studies, availability of discriminatory performance metrics, and age- and sex-specific data.

Results: In total, 903 FDA-approved AI-enabled medical devices were analyzed, most of which became available in the last decade. The devices primarily related to the specialties of radiology (692 devices [76.6.%]), cardiovascular medicine (91 devices [10.1%]), and neurology (29 devices [3.2%]). Most devices were software only (664 devices [73.5%]), and only 6 devices (0.7%) were implantable. Detailed descriptions of development were absent from most publicly provided summaries. Clinical performance studies were reported for 505 devices (55.9%), while 218 devices (24.1%) explicitly stated no performance studies were conducted. Retrospective study designs were most common (193 studies [38.2%]), with only 41 studies (8.1%) being prospective and 12 studies (2.4%) randomized. Discriminatory performance metrics were reported in 200 of the available summaries (sensitivity: 183 devices [36.2%]; specificity: 176 devices [34.9%]; area under the curve: 82 devices [16.2%]). Among clinical studies, less than one-third provided sex-specific data (145 studies [28.7%]), and only 117 studies (23.2%) addressed age-related subgroups.

Conclusions and relevance: In this cross-sectional study, clinical performance studies at the time of approval were reported for approximately half of AI-enabled medical devices, yet the information was often insufficient for a comprehensive assessment of their clinical generalizability, emphasizing the need for ongoing monitoring and regular re-evaluation to identify and address unexpected performance changes during broader use.

Figure 1.. Total Number of Available FDA-Approved AI-Enabled Medical Devices by Country and Percentage of Recalled Devices After FDA-Approval by Country

AI indicates artificial intelligence; FDA, Food and Drug Administration.

Figure 2.. Number of Artificial Intelligence–Enabled Medical Devices by Specialty, Along With Details on the Design of Their Clinical Performance Studies

Figure 3.. Performance Distribution and Associations Among AUC, Sensitivity, and Specificity Metrics for AI-Enabled Medical Devices in Clinical Performance Studies

AUC indicates area under the curve. The diagonal plots display the distribution of each individual metric (AUC, sensitivity, and specificity), while the off-diagonal scatter plots show the pairwise relationships between these metrics. Only 3 recalled devices had available sensitivity and specificity values, but none of them had AUC values.

Figure 4.. Time Lag Between Approval and Recall of the Corresponding Artificial Intelligence–Enabled Medical Devices

Devices are shortlisted based on their Food and Drug Administration approval timeline. Each device is identified by its unique Food and Drug Administration submission number.