Evaluation of research in biomedical ontologies

Abstract

Ontologies are now pervasive in biomedicine, where they serve as a means to standardize terminology, to enable access to domain knowledge, to verify data consistency and to facilitate integrative analyses over heterogeneous biomedical data. For this purpose, research on biomedical ontologies applies theories and methods from diverse disciplines such as information management, knowledge representation, cognitive science, linguistics and philosophy. Depending on the desired applications in which ontologies are being applied, the evaluation of research in biomedical ontologies must follow different strategies. Here, we provide a classification of research problems in which ontologies are being applied, focusing on the use of ontologies in basic and translational research, and we demonstrate how research results in biomedical ontologies can be evaluated. The evaluation strategies depend on the desired application and measure the success of using an ontology for a particular biomedical problem. For many applications, the success can be quantified, thereby facilitating the objective evaluation and comparison of research in biomedical ontology. The objective, quantifiable comparison of research results based on scientific applications opens up the possibility for systematically improving the utility of ontologies in biomedical research.

Keywords: biomedical ontology; evaluation criteria; ontology evaluation; ontology-based applications; quantitative biology.

Figure 1:

A direct evaluation of an ontology can assess intrinsic properties of the ontology such as consistency, expressivity, or the inclusion of natural language definitions and labels. Furthermore, the evaluating person can examine definitions and axioms of the ontology and either agree or disagree with their content.

Figure 2:

An application-based evaluation does not directly assess an ontology, but rather evaluates an application that utilizes an ontology for its operations.

Figure 3:

An analysis-based evaluation performs a scientific data analysis that relies on an ontology and evaluates the success of the analysis using criteria established in the scientific domain.