Evaluate the performance of four artificial intelligence-aided diagnostic systems in identifying and measuring four types of pulmonary nodules

Abstract

Purpose: This study aims to evaluate the performance of four artificial intelligence-aided diagnostic systems in identifying and measuring four types of pulmonary nodules.

Methods: Four types of nodules were implanted in a commercial lung phantom. The phantom was scanned with multislice spiral computed tomography, after which four systems (A, B, C, D) were used to identify the nodules and measure their volumes.

Results: The relative volume error (RVE) of system A was the lowest for all nodules, except for small ground glass nodules (SGGNs). System C had the smallest RVE for SGGNs, -0.13 (-0.56, 0.00). In the Bland-Altman test, only systems A and C passed the consistency test, P = 0.40. In terms of precision, the miss rate (MR) of system C was 0.00% for small solid nodules (SSNs), ground glass nodules (GGNs), and solid nodules (SNs) but 4.17% for SGGNs. The comparable system D MRs for SGGNs, SSNs, and GGNs were 71.30%, 25.93%, and 47.22%, respectively, the highest among all the systems. Receiver operating characteristic curve analysis indicated that system A had the best performance in recognizing SSNs and GGNs, with areas under the curve of 0.91 and 0.68. System C had the best performance for SGGNs (AUC = 0.91).

Conclusion: Among four types nodules, SGGNs are the most difficult to recognize, indicating the need to improve higher accuracy and precision of artificial systems. System A most accurately measured nodule volume. System C was most precise in recognizing all four types of nodules, especially SGGN.

Keywords: artificial intelligence; lung phantom; pulmonary nodules.

Fig. 1

(a) The image of the professional phantom; (b) The diagram of nodules. Nodules are randomly distributed in the phantom. −800HU, −630HU, and + 100HU were the density of nodules. And 3, 5, 8, 10, 12mm were the diameter of nodules.

Fig. 2

CT images of different types of nodules in phantom.

Fig. 3

RVE of each AIADS for detecting different nodules. RVE, the relative volume error; AIADS, artificial intelligence‐aided diagnosis system; SGGN, small ground glass nodule; SSN, small solid nodule; GGN, ground glass nodule; SN, solid nodule; A,B,C,D is the code name of four AIADSs.

Fig. 4

Consistency Test between system A and others. A,B,C,D is the code of four AIADSs.

Fig. 5

The one vs all ROC curves. (1) Dichotomies of SGGN and others; (2) Dichotomies of SSN and others; (3) Dichotomies of GGN and others; (4) Dichotomies of SN and others; A,B,C,D is the code of four AIADSs.