Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2008 Sep 18;60(1):123-134.
doi: 10.1002/asi.20955.

Natural Language Processing Versus Content-Based Image Analysis for Medical Document Retrieval

Aurélie Névéol  1 , Thomas M Deserno, Stéfan J Darmoni, Mark Oliver Güld, Alan R Aronson
Affiliations

Natural Language Processing Versus Content-Based Image Analysis for Medical Document Retrieval

Aurélie Névéol et al. J Am Soc Inf Sci Technol. .

Abstract

One of the most significant recent advances in health information systems has been the shift from paper to electronic documents. While research on automatic text and image processing has taken separate paths, there is a growing need for joint efforts, particularly for electronic health records and biomedical literature databases. This work aims at comparing text-based versus image-based access to multimodal medical documents using state-of-the-art methods of processing text and image components. A collection of 180 medical documents containing an image accompanied by a short text describing it was divided into training and test sets. Content-based image analysis and natural language processing techniques are applied individually and combined for multimodal document analysis. The evaluation consists of an indexing task and a retrieval task based on the "gold standard" codes manually assigned to corpus documents. The performance of text-based and image-based access, as well as combined document features, is compared. Image analysis proves more adequate for both the indexing and retrieval of the images. In the indexing task, multimodal analysis outperforms both independent image and text analysis. This experiment shows that text describing images can be usefully analyzed in the framework of a hybrid text/image retrieval system.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Sample document in the test corpus. An English translation of the text is provided by the authors for illustration purposes: Caption: “François, 1 1/2 month: Abdomen without preparation.” Paragraph: “Based on the child’s age and the fact that he experienced projectile vomiting, a diagnosis of pyloric stenosis can be made. An emergency abdomen without preparation is ordered for François; no evidence of gastric dilatation is visible.”
Figure 2
Figure 2
Processing multimodal biomedical documents for information retrieval.
Figure 3
Figure 3
Set of images in the test corpus matching the query “1121-115-700-400”.
Figure 4
Figure 4
Text and image analysis performed to assign IRMA codes to a test document. Text analysis (solid line and box) involved applying a dictionary and using the training collection to retrieve the 5 nearest neighbors (5-NN) for the text portion of a query document. The image analysis (dashed lines and boxes) used either the training collection or the IRMA database to retrieve the k nearest neighbors (k-NN) based on content-based image retrieval (CBIR) features.
Figure 5
Figure 5
Sample IRMA-MeSH equivalences made into dictionary entries (using MeSH hierarchy).
Figure 6
Figure 6
Sample IRMA-MeSH equivalences made into dictionary entries (adapting to IRMA specificity).
See this image and copyright information in PMC

References

    1. Aachen University of Technology Image retrieval in medical applications. 2008. Retrieved March 28, 2008, from http://www.irma-project.org/index_en.php.
    1. Aronson AR, Bodenreider O, Demner-Fushman D, Fung KW, Lee VK, Mork JG, et al. From indexing the biomedical literature to coding clinical text : Experience with MTI and machine learning approaches. In: Cohen K, Demner-Fushman D, Friedman C, Hirschman L, Pestian J, editors. Proceedings of the ACL 2007 Workshop on Biological, Translational, and Clinical Language Processing (BioNLP); 2007. pp. 105–112.
    1. Boujemaa N, Fauqueur J, Gouet V. IAPR International Conference on Image and Signal Processing (ICISP’ 2003), Agadir, Morocco. What's beyond query by example? 2003. Jun, Retrieved August 29, 2008, from http://www-rocq.inria.fr/~gouet/Recherche/Papiers/icisp03/pdf.
    1. Briet S. Qu’est-ce que la documentation? [What is documentation?] Paris: Éditions Documentaires et Industrielles; 1951. [English translation retrieved April 29, 2007, from http://martinetl.free.fr/suzanne_briet.htm and archived at http://www.webcitation.org/5OUOoaTjW]
    1. Byrne K, Klein E. Image retrieval using natural language and content-based techniques. In: de Vries AP, editor. Proceedings of the Fourth Dutch-Belgian Information Retrieval Workshop (DIR 2003); Amsterdam: Institute for Logic, Language, and Computation; 2003. pp. 57–62.

LinkOut - more resources