The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans

Abstract

Purpose: The development of computer-aided diagnostic (CAD) methods for lung nodule detection, classification, and quantitative assessment can be facilitated through a well-characterized repository of computed tomography (CT) scans. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) completed such a database, establishing a publicly available reference for the medical imaging research community. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process.

Methods: Seven academic centers and eight medical imaging companies collaborated to identify, address, and resolve challenging organizational, technical, and clinical issues to provide a solid foundation for a robust database. The LIDC/IDRI Database contains 1018 cases, each of which includes images from a clinical thoracic CT scan and an associated XML file that records the results of a two-phase image annotation process performed by four experienced thoracic radiologists. In the initial blinded-read phase, each radiologist independently reviewed each CT scan and marked lesions belonging to one of three categories ("nodule > or =3 mm," "nodule <3 mm," and "non-nodule > or =3 mm"). In the subsequent unblinded-read phase, each radiologist independently reviewed their own marks along with the anonymized marks of the three other radiologists to render a final opinion. The goal of this process was to identify as completely as possible all lung nodules in each CT scan without requiring forced consensus.

Results: The Database contains 7371 lesions marked "nodule" by at least one radiologist. 2669 of these lesions were marked "nodule > or =3 mm" by at least one radiologist, of which 928 (34.7%) received such marks from all four radiologists. These 2669 lesions include nodule outlines and subjective nodule characteristic ratings.

Conclusions: The LIDC/IDRI Database is expected to provide an essential medical imaging research resource to spur CAD development, validation, and dissemination in clinical practice.

Figure 1

Examples of lesions considered to satisfy the LIDC∕IDRI definition of (a) a nodule≥3 mm, (b) a nodule<3 mm, and (c) a non-nodule≥3 mm (reprinted with permission from Ref. 29).

Figure 2

(a) A lesion considered to be a nodule≥3 mm by all four LIDC∕IDRI radiologists. (b) The nested outline of one radiologist reflects the radiologist’s opinion that a region of exclusion (a dilated bronchus) exists within the nodule. The inner outline is explicitly noted as an exclusion in the XML file. Each outline is an “outer border” so that neither outline is meant to overlap pixels interpreted as belonging to the nodule.

Figure 3

(a) A lesion considered to be a nodule≥3 mm by two LIDC∕IDRI radiologists and a nodule<3 mm or non-nodule≥3 mm by the other two radiologists. (b) A lesion identified as a nodule≥3 mm (arrow) by three LIDC∕IDRI radiologists but assigned no mark at all by the fourth radiologist (reprinted with permission from Ref. 36).

Figure 4

Distributions depicting the proportions of the 7371 nodules that were (1) marked as a nodule by different numbers of radiologists (gray) or (2) assigned any mark at all (including non-nodule≥3 mm) by different numbers of radiologists (black).

Figure 5

Distributions depicting the proportions of the 2669 lesions marked by at least one radiologist as a nodule≥3 mm that were marked as such by different numbers of radiologists.

Figure 6

Examples of lesions marked as a nodule≥3 mm (a) by only a single radiologist (the other three radiologists identified this lesion as a non-nodule≥3 mm) and (b) by all four radiologists.

Figure 7

Distributions depicting the proportions of the 2669 lesions marked by at least one radiologist as a nodule≥3 mm that were marked as either a nodule≥3 mm or a nodule<3 mm by different numbers of radiologists.

Figure 8

(a) A lesion identified by three radiologists as a single nodule≥3 mm that was considered to be two separate nodules≥3 mm by the fourth radiologist. [(b) and (c)] The outlines constructed on this section by two of the radiologists.

Figure 9

A lesion identified by one radiologist as a single nodule≥3 mm that was considered to be a nodule≥3 mm (arrowhead) and a separate nodule<3 mm (arrow) by another radiologist and a non-nodule≥3 mm (arrowhead) and a separate nodule<3 mm (arrow) by two other radiologists.

Figure 10

Examples of differences in radiologists’ interpretation of nodule≥3 mm boundaries. (a) In-plane outlines differ between two radiologists in a single CT section. (b) A lesion depicted in two adjacent CT sections that is outlined by all four radiologists in the more superior section (left) but only by two radiologists in the more inferior section (right) (outlines not shown). (c) A nodule outline for which a portion (arrow) encloses no nodule pixels based on the outer border definition.