A method for evaluating the performance of computer-aided detection of pulmonary nodules in lung cancer CT screening: detection limit for nodule size and density

Abstract

Objective: We propose the application of virtual nodules to evaluate the performance of computer-aided detection (CAD) of lung nodules in cancer screening using low-dose CT.

Methods: The virtual nodules were generated based on the spatial resolution measured for a CT system used in an institution providing cancer screening and were fused into clinical lung images obtained at that institution, allowing site specificity. First, we validated virtual nodules as an alternative to artificial nodules inserted into a phantom. In addition, we compared the results of CAD analysis between the real nodules (n = 6) and the corresponding virtual nodules. Subsequently, virtual nodules of various sizes and contrasts between nodule density and background density (ΔCT) were inserted into clinical images (n = 10) and submitted for CAD analysis.

Results: In the validation study, 46 of 48 virtual nodules had the same CAD results as artificial nodules (kappa coefficient = 0.913). Real nodules and the corresponding virtual nodules showed the same CAD results. The detection limits of the tested CAD system were determined in terms of size and density of peripheral lung nodules; we demonstrated that a nodule with a 5-mm diameter was detected when the nodule had a ΔCT > 220 HU.

Conclusion: Virtual nodules are effective in evaluating CAD performance using site-specific scan/reconstruction conditions. Advances in knowledge: Virtual nodules can be an effective means of evaluating site-specific CAD performance. The methodology for guiding the detection limit for nodule size/density might be a useful evaluation strategy.

Figure 1.

A schematic explanation of virtual nodule generation: (a) the object function of a typical solitary pulmonary nodule with a diameter of 6 mm; (b) a computer-simulated nodule obtained from the object function by Equation (1); (c) a virtual nodule generated by resampling the previous image (b) in three dimensions at clinical CT image resolution; (d) a virtual nodule added to the clinical CT image (arrow).

Figure 2.

An image of a chest phantom including artificial nodules: there are five high-contrast nodules in the left lung with diameters of 2 mm, 4 mm, 6 mm, 8 mm and 10 mm (arrows) and five low-contrast nodules in the right lung with diameters of 12 mm, 10 mm, 8 mm, 6 mm and 4 mm (arrowheads). The high-contrast 2-mm nodule and the low-contrast 4-mm nodule were not used in this study because of the difficulty in identifying them in this image.

Figure 3.

Virtual nodules added to the phantom image containing artificial nodules: each virtual nodule (arrows) has been placed near the location of the corresponding artificial nodule (arrowheads).

Figure 4.

Real nodules (arrows) in patient images (a), (c) and (e) are showing computer-aided detection results of true positives (TPs) and corresponding virtual nodules (arrowheads) added into comparable images (b), (d) and (f) of other cases; images were generated using maximum intensity projection of three consecutive sections with the nodule in the centre section. The diameters and contrasts (between nodule density and background density) of object functions used for generating virtual nodules were 6.0 mm and 250 HU (b), 5.5 mm and 350 HU (d) and 5.6 mm and 310 HU (f).

Figure 5.

The analogous comparison of real nodules (left) with corresponding virtual nodules (right) as described in Figure 4, but here with real nodules showing computer-aided detection results of false negatives (FNs): the diameters and contrasts between nodule density and background density of object functions used for generating virtual nodules were 4.5 mm and 360 HU (b), 7.0 mm and 480 HU (d) and 4.3 mm and 320 HU (f).

Figure 6.

Five virtual nodules have been added to a clinical image at locations in the lung periphery: the object function diameters and contrasts (between nodule density and background density) are (a) 5 mm and 200 HU and (b) 7 mm and 400 HU, respectively.

Figure 7.

A comparison of virtual nodules with the phantom image containing high-contrast artificial nodules: (a) artificial nodules—the regions of interest (ROIs), indicated by boxes, were used to determine a representative standard deviation (SD) of the background intensity; (b) virtual nodules are corresponding to artificial nodules; (c) the image obtained by subtracting each nodule image (b) from (a). The SDs of the subtraction residuals were evaluated in the ROIs indicated by the dashed circles. A narrow window width (400 HU) was used for all images.

Figure 8.

Virtual nodules (arrows) not detected by the computer-aided detection (CAD) system: the virtual nodules shown in (a)–(c) are the same as those in Figure 5b,d,f respectively. The locations of the nodule centres were changed from their initial locations to those locations marked with closed triangles “▲” and closed circles “●” in the figure. When nodules were placed on the locations “▲”, they were not detected by the CAD system. When nodules were placed on the locations “●”, they were detected.

Figure 9.

Results of computer-aided detection (CAD) system detections of virtual nodules fused into clinical lung images: (a) two of the five virtual nodules were detected (indicated by boxes) when the object function was 5 mm in diameter and the contrast between nodule density and background density was (ΔCT) = 200 HU. (b) All five 5-mm virtual nodules were detected when the contrast was ΔCT = 300 HU. (c) The summary of all detection results with the detection limit is indicated by the solid line. When four or more nodules were detected (i.e. detection rate ≥80%), the data with the corresponding diameter and ΔCT in the figure are marked as “○” (detectable); otherwise, they are marked as “●” (undetectable).

Figure 10.

The detection limit of the computer-aided detection system averaged over all cases (n = 10) (Figure 9c). Error bars are indicating standard deviation. ΔCT, contrast between nodule density and background density.