Evaluation of the quality of information retrieval of clinical findings from a computerized patient database using a semantic terminological model

Abstract

Objectives: To measure the strength of agreement between the concepts and records retrieved from a computerized patient database, in response to physician-derived questions, using a semantic terminological model for clinical findings with those concepts and records excerpted clinically by manual identification. The performance of the semantic terminological model is also compared with the more established retrieval methods of free-text search, ICD-10, and hierarchic retrieval.

Design: A clinical database (Diabeta) of 106,000 patient problem record entries containing 2,625 unique concepts in an clinical academic department was used to compare semantic, free-text, ICD-10, and hierarchic data retrieval against a gold standard in response to a battery of 47 clinical questions.

Measurements: The performance of concept and record retrieval expressed as mean detection rate, positive predictive value, Yates corrected and Mantel-Haenszel chi-squared values, and Cohen kappa value, with significance estimated using the Mann-Whitney test.

Results: The semantic terminological model used to retrieve clinically useful concepts from a patient database performed well and better than other methods, with a mean detection rate of 0.86, a positive predictive value of 0.96, a Yates corrected chi-squared value of 1,537, a Mantel-Haenszel chi-squared value of 19,302, and a Cohen kappa of 0.88. Results for record retrieval were even better, with a mean record detection rate of 0.94, a positive predictive value of 0.99, a Yates corrected chi-squared value of 94, 774, a Mantel-Haenszel chi-squared value of 1,550,356, and a Cohen kappa value of 0.94. The mean detection rate, Yates corrected chi-squared value, and Cohen kappa value for semantic retrieval were significantly better than for the other methods.

Conclusion: The use of a semantic terminological model in this test scenario provides an effective framework for representing clinical finding concepts and their relationships. Although currently incomplete, the model supports improved information retrieval from a patient database in response to clinically relevant questions, when compared with alternative methods of analysis.

Figure 1

Concepts in Clinical Terms Version 3 are placed in a pure subtype hierarchy. The structure also allows the formal definition of concepts according their meaning (semantic definition); thus, Bacterial meningitis is represented by [Site]: Meninges; [Pathological process]: Infection; [Causative agent]: Bacteria. Note: The triangle represents a subtype relationship utilizing the notation of the Unified Modelling Language (UML), the Object Management Group (OMG) industry standard (Rational Software Corporation, Cupertino, California, 1995).

Figure 2

Abridged representation of the provisional semantic terminological model identifying the main characteristics of clinical findings expressed as attributes and applicable value concept hierarchies, with examples. The section mark (§) indicates that the expression of laterality is applied via anatomy.

Figure 3

Relationship between the four methods of retrieval, the original Diabeta clinical finding term string and the Clinical Terms Version 3 concepts (with their mapping to ICD-10, hierarchical position represented in the hierarchy table, and semantic definition) in the experimental database.

Figure 4

An extract of the semantic definition table from the experimental database, illustrating its use of a separate column for each attribute. The atoms for each concept are indicated by an entry of the appropriate value in the applicable attribute field. (Only a subset of attributes is shown.)

Figure 5

The specification of a query to retrieve all disorders affecting the limb includes those disorder concepts having a semantic definition [Site]: Limb structure or part of limb structure. (An extract from the anatomy value limb structure hierarchy is illustrated.)

Figure 6

The specification of a query to retrieve all disorders caused by infection and affecting the limbs includes those concepts having both a semantic definition of [Pathological process]: Infection and [Site]: Limb structure or part of limb structure. (An extract from the anatomy value limb structure hierarchy is illustrated.)

Figure 7

Venn diagram showing the relationship between the population of concepts observed by data retrieval (C^O), with respect to the actual (gold standard) expected population (C^E), and the total population (N); and the true positive, false positive, false negative and true negative populations forming the 2 × 2 contingency tables for each question in the experimental database.

Figure 8