Assessment of performance and reproducibility of applying a content-based image retrieval scheme for classification of breast lesions

Abstract

Purpose: To develop a new computer-aided diagnosis (CAD) scheme using a content-based image retrieval (CBIR) approach for classification between the malignant and benign breast lesions depicted on the digital mammograms and assess CAD performance and reproducibility.

Methods: An image dataset including 820 regions of interest (ROIs) was used. Among them, 431 ROIs depict malignant lesions and 389 depict benign lesions. After applying an image preprocessing process to define the lesion center, two image features were computed from each ROI. The first feature is an average pixel value of a mapped region generated using a watershed algorithm. The second feature is an average pixel value difference between a ROI's center region and the rest of the image. A two-step CBIR approach uses these two features sequentially to search for ten most similar reference ROIs for each queried ROI. A similarity based classification score was then computed to predict the likelihood of the queried ROI depicting a malignant lesion. To assess the reproducibility of the CAD scheme, we selected another independent testing dataset of 100 ROIs. For each ROI in the testing dataset, we added four randomly queried lesion center pixels and examined the variation of the classification scores.

Results: The area under the ROC curve (AUC) = 0.962 ± 0.006 was obtained when applying a leave-one-out validation method to 820 ROIs. Using the independent testing dataset, the initial AUC value was 0.832 ± 0.040, and using the median classification score of each ROI with five queried seeds, AUC value increased to 0.878 ± 0.035.

Conclusions: The authors demonstrated that (1) a simple and efficient CBIR scheme using two lesion density distribution related features achieved high performance in classifying breast lesions without actual lesion segmentation and (2) similar to the conventional CAD schemes using global optimization approaches, improving reproducibility is also one of the challenges in developing CAD schemes using a CBIR based regional optimization approach.

FIG. 1.

Illustration of 50 malignant ROIs extracted in our independent testing dataset.

FIG. 2.

Illustration of 50 benign ROIs extracted in our independent testing dataset.

FIG. 3.

An example of a matrix extracted from “a lesion center” (a) and the corresponding output of applying watershed function processing (b).

FIG. 4.

A diagram showing the variation of AUC values versus the increase of kernel size of a watershed matrix to compute feature 1 (F₁).

FIG. 5.

A scatter diagram showing the distribution of two image features computed from 820 ROIs. The red square marks indicate malignant ROIs, and the blue diamond marks represent benign ROIs.

FIG. 6.

Histograms of the number of malignant reference ROIs retrieved using feature 1 (F₁) among the 50 malignant and 50 benign testing ROIs.

FIG. 7.

Comparison of three ROC curves generated using the rockit program. The computed AUC values are 0.962 ± 0.006, 0.603 ± 0.020, and 0.515 ± 0.020 when using our CBIR based classification scores (a leave-one-ROI-out based validation method), the average of two image features, and the first feature computed from the watershed algorithm generated maps, respectively.

FIG. 8.

Distribution of classification scores computed using the original queried lesion center seeds (marked by *) and the median classification scores (marked by ^∘) among 50 malignant ROIs (a) and 50 benign ROIs (b). In both diagrams (a) and (b), the solid red line and dashed blue line indicate the average classification score level of the 50 ROIs using the original seeds and median classification scores using the five seeds, respectively.

FIG. 9.

Comparison of two ROC curves generated using two different sets of classification scores of 100 testing ROIs. In these two ROC curves, AUC = 0.832 ± 0.040 and 0.878 ± 0.035 when using the classification scores computed based on the original queried lesion center seed and the median classification scores computed from five randomly placed lesion center seeds, respectively.