A Semantic Similarity Measure in Document Databases: An Earth Mover's Distance-Based Approach

Abstract

Measuring document similarity is important in order to find documents which are similar to a given query document from a user. Text-based document similarity is measured by comparing the words in two documents. The representative text-based document similarity is the cosine similarity. Since the cosine similarity computes document similarity by estimating the frequency of common words, it cannot reflect word similarity. To solve this problem, we propose a new document similarity measure based on the earth mover's distance (EMD). The EMD is one of the most popular distance functions used to search similar multimedia contents and is known to provide good search results. To apply the EMD to compute document similarity, we have to solve two problems: (1) The EMD is too time consuming to be used in a document database, (2) the distance between words should be defined. Our proposed approach first extracts topics as new features of a document by applying the latent Dirichlet allocation, which is a generative model of a document. It can decrease the computational cost of the EMD because the number of topics is much smaller than the number of words in a document. After extracting the topics, the proposed approach calculates the distance between topics based on the relation between the topics and the words in a document database, thereby making computing document similarity based on the EMD possible. Our approach searches documents more accurately since we can consider the semantic similarity by using the EMD. Experimental results on a real-world document database indicate that the proposed approach outperforms the cosine similarity in terms of the accuracy and the performance.