Annotation of enhanced radiographs for medical image retrieval with deep convolutional neural networks

Abstract

The number of images taken per patient scan has rapidly increased due to advances in software, hardware and digital imaging in the medical domain. There is the need for medical image annotation systems that are accurate as manual annotation is impractical, time-consuming and prone to errors. This paper presents modeling approaches performed to automatically classify and annotate radiographs using several classification schemes, which can be further applied for automatic content-based image retrieval (CBIR) and computer-aided diagnosis (CAD). Different image preprocessing and enhancement techniques were applied to augment grayscale radiographs by virtually adding two extra layers. The Image Retrieval in Medical Applications (IRMA) Code, a mono-hierarchical multi-axial code, served as a basis for this work. To extensively evaluate the image enhancement techniques, five classification schemes including the complete IRMA code were adopted. The deep convolutional neural network systems Inception-v3 and Inception-ResNet-v2, and Random Forest models with 1000 trees were trained using extracted Bag-of-Keypoints visual representations. The classification model performances were evaluated using the ImageCLEF 2009 Medical Annotation Task test set. The applied visual enhancement techniques proved to achieve better annotation accuracy in all classification schemes.

Fig 1. Example of two grayscale radiographs annotated with the 13-digit classification code.

Both images were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 2. Examples of radiographs annotated with two classes from the T-scheme.

(A) shows three images belonging to class ‘1124’ representing ‘Xray; Plain Radiology; Analog; Low Beam Energy’ and (B) displays three images belonging to class ‘1123’ representing ‘Xray; Plain Radiology; Analog; High Beam Energy’. All radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 3. Examples of radiographs annotated with two classes from the D-scheme.

(A) shows three image belonging to class ‘125’ representing ‘Coronal; Anteroposterior; Upright’ and (B) displays three images belonging to class ‘228’ representing ‘Sagital; Lateral, left-right; Inclination’. All radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 4. Examples of radiographs annotated with two classes from the A-scheme.

(A) shows three images each belonging to class ‘732’ representing ‘Abdomen; Lower abdomen; Lower middle quadrant’ and (B) displays three images belonging to ‘213’ representing ‘Cranium; Facial cranium; Nose area’. All radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 5. Examples of radiographs annotated with two classes from the B-scheme.

(A) shows three imagse belonging to class ‘443’ representing ‘Gastrointestinal system; Small intestine; Ileum’ and (B) displays three images belonging to class ‘512’ representing ‘Uropoietic system; Kidney; Renal pelvis’. All radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 6. Medical image before and after Contrast Limited Adaptive Histogram Equation (CLAHE) was performed.

The radiograph was randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from[RWTH Aachen], original copyright [2009].

Fig 7. Medical image before and after applying the Non Local Means (NL-MEANS) preprocessing method.

The radiograph was randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 8. Enhanced grayscale radiograph, by augmenting 2 extra color layers to obtain a RGB-channeled medical image.

The radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Republished from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 9. Resized radiographs by padding input images to the defined width and height size [512 x 512].

(A) shows horizontal and (B) vertical padding. The radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Modified from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].

Fig 10. Output image after successively applying the image padding and image layering enhancement techniques.

The radiographs were randomly chosen from the ImageCLEF 2009 Medical Annotation Task Training Set. Modified from [21] under a CC BY license, with permission from [RWTH Aachen], original copyright [2009].