Independent Evaluation of a Commercial AI Software for Incidental Findings of Pulmonary Embolism (IPE) on a Large Hospital Retrospective Dataset

Abstract

Background: Early treatment of pulmonary embolism is associated with better outcomes, yet incidental PE (IPE) is frequently missed. This retrospective study aims to provide an independent assessment an artificial intelligence (AI) software, developed for flagging IPEs on CT scans. Methods: The study included consecutive CT examinations of 5042 unique patients (8 scanners and 3 protocols) acquired at a large NHS Trust between 01 January 2022 and 30 September 2022. Two radiologists blindly and independently reviewed the AI "positive" and a random selection of "negative" cases to establish the reference standard (n = 200). Discrepancies were adjudicated by a third radiologist. The clinical reports of the 200 cases were reviewed for comparison. Performance metrics for the software were calculated for the full (n = 5042) and reviewed (n = 200) cohorts separately. Results: Based on the reference standard, the IPE prevalence was 1.6% (81/5041). Across the reviewed cohort, the algorithm detected PE with a sensitivity of 96.4%, a specificity of 89.7%, a PPV of 87.1%, an NPV of 97.2%, and an accuracy of 92.5%. Across the full cohort, the algorithm exhibited a sensitivity of 96.4%, a specificity of 99.8%, a PPV of 87.1%, an NPV of 99.9%, and an accuracy of 99.7%. A review of the original clinical reports indicated that 11 cases of IPE were initially unreported. A total of 34 examinations were rejected by the software. While the scanner performed consistently across patient sexes and ethnicities, discrepancies were found among CT scanners. Conclusions: The AI software detected IPE with a high diagnostic accuracy on a large NHS dataset, showing that AI-supported reporting could improve diagnostic accuracy and reduce times to diagnosis.

Figure 1

Flowchart of study inclusions and exclusions. AI = artificial intelligence; CT = computed tomography; CTPA = CT pulmonary angiogram.