Similar image search for histopathology: SMILY

Abstract

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.

Keywords: Image processing; Machine learning; Medical imaging; Software.

Fig. 1

Overview of Similar Medical Images Like Yours (SMILY). First, a database of image patches and a numerical characterization of each patch’s image contents (termed the embedding) is created. SMILY uses a convolutional neural network to compute this embedding (schematic used for illustration purposes only, see Methods for architecture descriptions). Next, when a query image is selected, SMILY computes the embedding of that query image and compares the embedding with those in the database in a computationally efficient manner. Finally, SMILY returns the k most similar patches, where k is customizable

Fig. 2

Sample view of the SMILY user interface. Sample query from a prostate specimen and search results. One of the search results has been magnified for better visualization. Clicking a search result opens a new viewer centered on the result that can be zoomed in for detail or zoomed out for context. Additional examples of queries and search results are presented in Supplementary Fig. 4, including an additional interface for scoring the quality of each search result for the prospective studies with pathologists

Fig. 3

SMILY search accuracy from large-scale quantitative evaluation using pathologist-provided annotations. a Results for histologic feature match in prostate specimens, in comparison with a traditional image feature extractor (scale-invariant feature transform, SIFT) and random search. b Results for prostate cancer Gleason grade and histologic feature match, in comparison with the same baselines. Error bars indicate 95% confidence intervals (Methods)

Fig. 4

Confusion matrix from SMILY search. An element in row i, column j indicates the fraction of search results for query i that result in a “hit” based on the top-5 score for the category j. a Confusion matrix for the results from Fig. 3a: histologic feature match in prostate specimens. b Confusion matrix for histologic feature match across prostate, breast, and colon specimens. To improve visual contrast and highlight trends better, only discrete colors and rounded-off values are used

Fig. 5

Evaluation of SMILY from studies with pathologists. The pathologists evaluated search results, blinded to whether the results were retrieved by SMILY versus a negative control, random selection. The similarity scoring rubrics are detailed in the “Prospective studies with pathologists” subsection in the Methods. a Histologic feature match in prostate specimens. b Histologic feature match in prostate, breast, and colon specimens. c Organ site match in prostate, breast, and colon specimens. d Overall match score (Table 3) in prostate specimens for similarity in histology and prostate cancer Gleason grade. Error bars indicate 95% confidence intervals (Methods)

Fig. 6

Visualizations of the embeddings of image patches in the SMILY database. Each dot represents an image patch. a Colored by organ site, indicating that patches from the same organ were distributed among different clusters. b Colored by histologic feature, indicating a more distinct separation between histologic features