The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans

Abstract

Rationale and objectives: The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured.

Materials and methods: Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule <3 mm," or "non-nodule >or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.

Results: After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.

Conclusion: The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.

Figure 1

(a) A region from one transaxial CT section. (b) The assigned unblinded read marks. A hexagon indicates an assigned “nodule < 3 mm” mark, while contours constructed by a radiologist indicate a “nodule ≥ 3 mm” designation. Based on a manual assessment of the three-dimensional contiguity of lesions and marks in the scan, these five marks were grouped into two distinct lesions: the upper lesion with one radiologist’s “nodule < 3 mm” mark and another radiologist’s “nodule ≥ 3mm” mark, and the lower lesion with one radiologist’s “nodule < 3 mm” mark and two radiologists’ “nodule ≥ 3mm” marks.

Figure 2

Number of lesions marked by at least one radiologist during the blinded and unblinded reading sessions.

Figure 3

(a) The single lesion that was marked only during the unblinded reading phase and (b) a lesion that was marked only during the blinded reading phase. The lesion in (a) was not marked by any radiologists during the blinded reading phase, but was marked as a “non-nodule” by one radiologist during the unblinded reading phase. The lesion in (b) was marked as a “micronodule” by a single radiologist during the blinded reading phase, but was not marked by any radiologist during the unblinded reading phase.

Figure 4

The distribution of the number of radiologists (out of a maximum of 4) marking individual lesions (from among the 466 lesions identified by at least one radiologist in this study) during the blinded and unblinded reads. Blinded-read-only lesions and unblinded-read-only lesions appear at 0 along the x-axis.

Figure 5

The distribution of change in the number of radiologists marking individual lesions between blinded and unblinded reads, and the distribution of change in the number of lesion categories to which those marks were assigned for individual lesions between the blinded and unblinded reads. While the mode of both distributions is 0, the trend toward more radiologists marking lesions in the unblinded reads (right skew of the radiologist distribution) and the trend toward those marks spanning fewer lesion categories (left skew of the category distribution) are evident.

Figure 6

Examples of lesions for which radiologists changed their assigned lesion category between the blinded and unblinded reading sessions either to or from the “nodule” category. The categories of one radiologist are reported for each lesion. (a) A blinded read “nodule” by one radiologist that was changed to “micronodule” during the unblinded read of that same radiologist. (b) A blinded read “nodule” that was changed to “non-nodule” during the unblinded read. (c) A blinded read “nodule” that received no mark at all during the unblinded read. (d) A blinded read “micronodule” that was changed to “nodule” during the unblinded read. (e) A blinded read “non-nodule” that was changed to “nodule” during the unblinded read. (f) A lesion that did not receive any mark during the blinded read of one radiologist that was assigned to the “nodule” lesion category during the unblinded read of that same radiologist.

Figure 7

Examples of lesions assigned to the “nodule” lesion category by different numbers of radiologists. (a) A lesion assigned to the “nodule” category by all four radiologists during both blinded and unblinded reading phases. (b) A lesion assigned to the “nodule” category by two radiologists during the blinded reads (the other two radiologists provided no mark during their blinded reads) and then by all four radiologists during the unblinded reads. (c) A lesion assigned to the “nodule” category by all four radiologists during the blinded reads but then by only three radiologists during the unblinded reads (the fourth radiologist assigned this lesion to the “non-nodule” category during the unblinded read). (d) A lesion assigned to the “nodule” category by a single radiologist during both blinded and unblinded reading phases (one of the other three radiologists assigned this lesion to the “micronodule” category during both blinded and unblinded reads, while the remaining two radiologists did not provide any marks during either reading phase). Each increment of the calibration scales corresponds to 1 mm.

Figure 8

A lesion that was assigned to the nodule category by a single radiologist during the blinded read (with no other marks placed by any other radiologist) and as a non-nodule by a different radiologist during the unblinded read (with no other marks placed by any other radiologist, including the radiologist who had marked the lesion during the blinded read).