Public evidence on AI products for digital pathology

Abstract

Novel products applying artificial intelligence (AI)-based methods to digital pathology images are touted to have many uses and benefits. However, publicly available information for products can be variable, with few sources of independent evidence. This review aimed to identify public evidence for AI-based products for digital pathology. Key features of products on the European Economic Area/Great Britain (EEA/GB) markets were examined, including their regulatory approval, intended use, and published validation studies. There were 26 AI-based products that met the inclusion criteria and, of these, 24 had received regulatory approval via the self-certification route as General in vitro diagnostic (IVD) medical devices. Only 10 of the products (38%) had peer-reviewed internal validation studies and 11 products (42%) had peer-reviewed external validation studies. To support transparency an online register was developed using identified public evidence ( https://osf.io/gb84r/ ), which we anticipate will provide an accessible resource on novel devices and support decision making.

Fig. 1. Conformity assessment of medical devices.

a Table comparing the relevant bodies involved in regulatory approval in the EEA and GB/Northern Ireland (NI), and the timeline for acceptance of compliant devices under different legislation. The EU MDR and EU IVDR entered into force in 2017 and became applicable in 2021 and 2022, respectively. For placement on the EEA market, there is a transitional period until December 2027 for higher risk devices (i.e., Class III and IIb implantable devices) and December 2028 for medium and lower risk devices (i.e., other Class IIb devices, Class IIa, Class Im, Is, and Ir devices). Subject to certain conditions (including an agreement in place with a notified body before certificate expiry), until those dates, certificates issued by a notified body under the old EU MDD, that were valid in May 2021, will remain valid. Additionally, Class I devices with an existing self-declaration of conformity (as of May 2021), which now require notified body involvement under the new EU MDR, can be placed on the market until December 2028. Self-certified Class I devices not requiring notified body involvement under EU MDR have no transitional period and must conform with EU MDR from May 2021. Similarly, subject to certain conditions, IVDs with a valid certificate from a notified body, issued under Annex VI of the IVDD before May 2022, can be placed on the market up until May 2025^,. For General IVDs with manufacturer self-declared conformity under the EU IVDD before May 2022, that now require notified body involvement under IVDR, the transition period is until May 2025 for higher risk Class D IVDs, May 2026 for Class C IVDs, and May 2027 for Class B and sterile Class A IVDs^,. Non-sterile Class A IVDs, which do not require notified body involvement under EU IVDR, and new IVDs, have no transitional period and must conform with the EU IVDR from May 2022. For placement on the GB market, medical devices with a valid certificate and CE-mark compliant with the EU MDD prior to May 2021 or EU IVDD prior to May 2022, can be placed on the market up until June 2028 or June 2030, respectively, as long as the certificate remains valid. Class I medical devices or General IVDs with self-declared conformity under EU MDD/EU IVDD, that will be upclassified under the new EU MDR/EU IVDR to require notified body involvement, can also continue to be placed on the GB market, subject to certain conditions, as long as the declaration remains valid. Additionally, devices with a valid CE-mark compliant with the EU MDR or EU IVDR can be placed on the GB market until June 2030. Products with a UKCA mark (currently conforming to the UK MDR 2002) can also be placed on the GB market. Note that Research Use Only (RUO) products are not subject to these regulatory standards as they are not for use in diagnostic procedures, and different requirements apply to in-house devices (i.e., those developed by, and for use only in, a specific institution) (see ref. ). Revised legislation is anticipated to apply in the UK beyond 2030. Under the terms of the Northern Ireland Protocol, different rules apply to devices placed on the market in Northern Ireland. See MHRA timeline for full details. Red arrow indicates increasing risk; 1m = Class 1 measuring function, 1s = Class 1 sterile, 1r = Class 1 reusable. Conformité Européene (CE). b Summary of the process and organisations involved in conformity assessment in the EEA and GB, for different device classes, under different legislation. In the EEA, conformity assessment bodies (Notified Bodies) are accredited by country-specific National Accreditation Bodies and designated by the Competent Authority for Medical Devices. For the GB market (defined as England, Scotland, and Wales, but excluding Northern Ireland; NI), conformity assessment bodies (Approved Bodies) can only provide UKCA product marking, with UKAS being the National Accreditation Body and the MHRA the Competent Authority. The Department for Business and Trade is responsible for appointing and monitoring UKAS.

Fig. 2. Product identification.

a Flow diagram indicating the number of companies and products identified during searches, including the total that met all inclusion criteria. b Pie chart showing the proportion of tools that met criteria for further analysis.

Fig. 3. Key features of eligible products.

Graphs showing (a) the size of companies producing eligible AI-based digital pathology products, (b) the type of regulatory approval received, (c) the pathology subspecialty, and (d) the primary purpose of each the product. In Vitro Diagnostic Device (IVD); Directive 98/79/EC on In Vitro Diagnostic Medical Devices (IVDD); Regulation (EU) 2017/746 on In Vitro Diagnostic Medical Devices (IVDR); UK Conformity Assessed (UKCA); United States Food and Drug Administration (FDA).

Fig. 4. Scientific publications associated with eligible products.

a Pie charts showing the proportion of products with accompanying scientific publications, b the type of study described, the accessibility of the publication, and the independence from vendors. c Graphs depicting the time between a product receiving regulatory approval and release of the first associated scientific publication, and d all scientific publications. Products with preprints are indicated in italics. Information was extracted from 23 identified publications, two of which described the validation of two separate products. Therefore 25 lines are displayed on the graph to indicate the publications related to the different products.

Fig. 5. Datasets used for validation studies.

a Number of whole slide images (WSIs) and cases/patients used for training and internal validation testing datasets across all studies. b Number of scanner platforms used to produce WSIs for internal validation testing data. c Pie chart showing the proportion of studies that included data from a UK-based institution in their internal validation. d World map displaying the country of origin of the data used for training and (e) testing for internal validation studies. f Number of WSIs and cases/patients used for external validation testing across all studies. g Number of scanner platforms used to produce WSIs for external validation testing data. h Pie chart showing the proportion of studies that included data from a UK-based institution in their external validation. i World map displaying the country of origin of the data used for external validation studies.