Lung Nodule Malignancy Prediction in Sequential CT Scans: Summary of ISBI 2018 Challenge

Abstract

Lung cancer is by far the leading cause of cancer death in the US. Recent studies have demonstrated the effectiveness of screening using low dose CT (LDCT) in reducing lung cancer related mortality. While lung nodules are detected with a high rate of sensitivity, this exam has a low specificity rate and it is still difficult to separate benign and malignant lesions. The ISBI 2018 Lung Nodule Malignancy Prediction Challenge, developed by a team from the Quantitative Imaging Network of the National Cancer Institute, was focused on the prediction of lung nodule malignancy from two sequential LDCT screening exams using automated (non-manual) algorithms. We curated a cohort of 100 subjects who participated in the National Lung Screening Trial and had established pathological diagnoses. Data from 30 subjects were randomly selected for training and the remaining was used for testing. Participants were evaluated based on the area under the receiver operating characteristic curve (AUC) of nodule-wise malignancy scores generated by their algorithms on the test set. The challenge had 17 participants, with 11 teams submitting reports with method description, mandated by the challenge rules. Participants used quantitative methods, resulting in a reporting test AUC ranging from 0.698 to 0.913. The top five contestants used deep learning approaches, reporting an AUC between 0.87 - 0.91. The team's predictor did not achieve significant differences from each other nor from a volume change estimate (p =.05 with Bonferroni-Holm's correction).

Fig. 1.

Representative patient cases showing 2D CT slice across two time points (t₁ & t₂) with a) Benign and b) Malignant diagnosis.

Fig. 2.

Challenge participant’s predictor performance. A) Receiver operator characteristic curve for participant predictors (11 teams). B) Bar graph showing the highest AUC values with 95% confidence limits computed by bootstrap resampling. Volume change predictor is shown for baseline comparison.

Fig. 3.

Boxplot show distribution of scores for cases and control (x-axis is the patient index, y-axis is the malignancy score) that helps to identify cases where most participants incorrectly reported the diagnostic status, a) Benign (see higher scores), b) Malignant (see lower scores) diagnosis. Representative patient cases showing 2D CT slice across two time points with most teams reporting incorrect malignancy scores. c) Benign diagnosis (average reported score 0.67) and d) Malignant diagnosis. (average reported score 0.26).