Ontology search: An empirical evaluation

Much of the recent work in Semantic Search is concerned with addressing the challenge of finding entities in the growing Web of Data. However, alongside this growth, there is a significant increase in the availability of ontologies that can be used to describe these entities. Whereas several methods have been proposed in Semantic Search to rank entities based on a keyword query, little work has been published on search and ranking of resources in ontologies. To the best of our knowledge, this work is the first to propose a benchmark suite for ontology search. The benchmark suite, named CBRBench1, includes a collection of ontologies that was retrieved by crawling a seed set of ontology URIs derived from prefix.cc and a set of queries derived from a real query log from the Linked Open Vocabularies search engine. Further, it includes the results for the ideal ranking of the concepts in the ontology collection for the identified set of query terms which was established based on the opinions of ten ontology engineering experts.

We compared this ideal ranking with the top-k results retrieved by eight state-of-the-art ranking algorithms that we have implemented and calculated the precision at k, the mean average precision and the discounted cumulative gain to determine the best performing ranking model. Our study shows that content-based ranking models outperform graph-based ranking models for most queries on the task of ranking concepts in ontologies. However, as the performance of the ranking models on ontologies is still far inferior to the performance of state-of-the-art algorithms on the ranking of documents based on a keyword query, we put forward four recommendations that we believe can significantly improve the accuracy of these ranking models when searching for resources in ontologies.