Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data

Abstract

Medical imaging is fundamental to modern healthcare, and its widespread use has resulted in the creation of image databases, as well as picture archiving and communication systems. These repositories now contain images from a diverse range of modalities, multidimensional (three-dimensional or time-varying) images, as well as co-aligned multimodality images. These image collections offer the opportunity for evidence-based diagnosis, teaching, and research; for these applications, there is a requirement for appropriate methods to search the collections for images that have characteristics similar to the case(s) of interest. Content-based image retrieval (CBIR) is an image search technique that complements the conventional text-based retrieval of images by using visual features, such as color, texture, and shape, as search criteria. Medical CBIR is an established field of study that is beginning to realize promise when applied to multidimensional and multimodality medical data. In this paper, we present a review of state-of-the-art medical CBIR approaches in five main categories: two-dimensional image retrieval, retrieval of images with three or more dimensions, the use of nonimage data to enhance the retrieval, multimodality image retrieval, and retrieval from diverse datasets. We use these categories as a framework for discussing the state of the art, focusing on the characteristics and modalities of the information used during medical image retrieval.

Fig. 1

A generic CBIR framework. The dashed arrows show the offline creation of the feature index from the image repository. The solid arrows show the online query process. The dashed line divides the offline and online processes. Note that feature extraction participates in both the offline and online processes

Fig. 2

A subset of the medical images available in many hospitals. Clockwise from the top left, they are axial CT slice, axial PET slice, axial fused PET-CT slice, coronal MR slice, and chest X-ray

Fig. 3

The graph representation used by Kumar et al. [64, 65] for PET-CT retrieval. a, c The CT and PET images acquired by the scanner, respectively; b the graph representing the relationships between the ROIs, including intermodality relationships between PET tumors and CT organs