Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation

Abstract

Automated computer-aided detection (CADe) has been an important tool in clinical practice and research. State-of-the-art methods often show high sensitivities at the cost of high false-positives (FP) per patient rates. We design a two-tiered coarse-to-fine cascade framework that first operates a candidate generation system at sensitivities ∼ 100% of but at high FP levels. By leveraging existing CADe systems, coordinates of regions or volumes of interest (ROI or VOI) are generated and function as input for a second tier, which is our focus in this study. In this second stage, we generate 2D (two-dimensional) or 2.5D views via sampling through scale transformations, random translations and rotations. These random views are used to train deep convolutional neural network (ConvNet) classifiers. In testing, the ConvNets assign class (e.g., lesion, pathology) probabilities for a new set of random views that are then averaged to compute a final per-candidate classification probability. This second tier behaves as a highly selective process to reject difficult false positives while preserving high sensitivities. The methods are evaluated on three data sets: 59 patients for sclerotic metastasis detection, 176 patients for lymph node detection, and 1,186 patients for colonic polyp detection. Experimental results show the ability of ConvNets to generalize well to different medical imaging CADe applications and scale elegantly to various data sets. Our proposed methods improve performance markedly in all cases. Sensitivities improved from 57% to 70%, 43% to 77%, and 58% to 75% at 3 FPs per patient for sclerotic metastases, lymph nodes and colonic polyps, respectively.

Fig. 1.

ConvNet applied to a 2.5D volume of interest extracted from a CT image. The number of convolutional filters, kernel sizes, and neural network connections for each layer are as shown. We use overlapping kernels with stride 2 during max-pooling.

Fig. 2.

Features are computed by convolving filter kernels over the input region of interest. The input image can be padded to produce convolution responses of the same size as the input image.

Fig. 3.

Some examples of filter responses (Right) after convolution with trained ConvNet kernels (Middle) of the first layer (showing an example of a sclerotic bone lesion in CT (Left).

Fig. 4.

CADe locations can be either observed as 2D image patches or using a 2.5D approach, that samples the image using three orthogonal views. Here, a lymph node in CT is shown as the input to our method.

Fig. 5.

Image patches are generated from CADe candidates using different scales, 2D/3D translations (along a random vector v) and rotations (by a random angle α) in the axial/3D plane (The example shows a sclerotic bone lesion in CT).

Fig. 6.

The first layer of 64 learned convolutional kernels of a ConvNet trained on medical CT images on each of three different CT imaging data sets: a) sclerotic metastases, b) lymph nodes and c) colonic polyps. The color coding in b) and c) illustrates the filters kernels used in each orthogonal view when using our 2.5D approach. The learned convolutional filters for sclerotic metastases in a) are using one-channel as input only (encoded in gray scale and learned from axial CT images). Here, complex higher order gradients, blobness and difference of Gaussian filters dominate. In b,c), the convolutional filters for lymph nodes or colonic polyps are three-channels (encoded in RGB and trained using three orthogonal CT views per example). Kernels learned from lymph nodes are mostly blobness and gradients of different orientations/channels in b). Colonic polyp kernels in c) are visually more diversified than the filters in b), especially with new “pointy” patterns probably resembling polyp intrusions from 3D colonic surfaces or tips.

Fig. 7.

Detection of sclerotic metastases: test probabilities of the ConvNet for being sclerotic metastases on ‘true’ sclerotic metastases candidate examples (1.0 equals 100% probability of representing a true positive) [21].

Fig. 8.

Detection of sclerotic metastases: test probabilities of the ConvNet for being sclerotic metastases on ‘false’ sclerotic metastases candidate examples (0.0 equals 100% probability of representing a false positive) [21].

Fig. 9.

Detection of sclerotic metastases: FROC curves for a 5-fold cross-validation using varying numbers of N random view ConvNet observations in testing of 59 patients (49 with sclerotic metastases and 10 normal controls). AUC values are computed for corresponding ROC curves [21].

Fig. 10.

Detection of sclerotic metastases: comparison of FROC curves of the initial bone lesion candidate generation (squares) compared to the final classification using N = 100 random view ConvNet observations (lines) for both training and testing cases. Results are computed using a 5-fold cross-validation in 59 patients (49 with sclerotic metastases and 10 normal controls) [21].

Fig. 11.

Detection of lymph nodes: test probabilities of the ConvNet for being a lymph node on ‘true’ (top box) and ‘false’ (bottom box) lymph node candidate examples [20].

Fig. 12.

Detection of lymph nodes: FROC curves for a 3-fold cross-validation using a varying number of N random view ConvNet observations in 176 patients. AUC values are computed for corresponding ROC curves. The previous performance by [25] is shown for comparison.

Fig. 13.

Comparison of the FROC performance of the previous method as candidate generation step using a random forest classifier [25] against an alternative second level classification approach using histogram of oriented gradients (HoG) [6] and the proposed 2.5D ConvNet approach using ConvNet observation on N = 100 random views.

Fig. 14.

Comparison of the FROC performance of when training a ConvNet with 2D, 2.5D and 3D inputs of the original (“ORIG”) or augmented (“AUG”) CT data. In the “ORIG” setting, 2D ConvNet shows the best generalized testing FROC result, followed by 3D and 2.5D ConvNets. The 2.5D approach using aggregation of random observations (“AUG”) in both training (Left) and testing (Right), out-performs both 2D and 3D approaches on the original data at the 3 FPs/patient level. The 2.5D ConvNet trained on augmented data overall performs comparably to a more computationally expensive 3D ConvNet approach on augmented 3D inputs. In brief, the evaluated 2.5D “AUG” ConvNet is chosen as the best trade-off lymph node detection model between effectiveness and efficiency.

Fig. 15.

Detection of colonic polyps: FROC curves for different polyp sizes, using up to N = 40 random view ConvNet observations in 792 testing CT colonography patients.