313 0

Document Similarity Measure Based on the Earth Mover's Distance Utilizing Latent Dirichlet Allocation

Title
Document Similarity Measure Based on the Earth Mover's Distance Utilizing Latent Dirichlet Allocation
Author
김상욱
Keywords
Cosine similairty; document similarity; earth mover’s distance; latent dirichlet allocation; semantic similarity
Issue Date
2016-01
Publisher
Medwell
Citation
Research Journal of Applied Sciences, Engineering & Technology, v. 12, NO 2, Page. 214-222
Abstract
Document similarity is used to search for such documents similar to a query document given. Text-based document similarity is computed by comparing the words in documents. The cosine similarity is the most popular text-based document similarity measure and computes the similarity of two documents based on their common word frequencies. It counts the exactly same words only, so cannot reflect semantic similarity between similar words having the same meaning. We propose a new document similarity measure to solve this problem by using the Earth Mover’s Distance (EMD). The EMD enables to compute the semantic similarity of documents. To apply the EMD to the similarity measure, we need to solve the high computational complexity and to define the distance between attributes. The high computational complexity comes from the large number of words in documents. Thus, we extract the topics from documents by using Latent Dirichlet Allocation (LDA), a document generating model. Since the number of topics is much smaller than that of words, the LDA helps reduce the computational complexity. We define the distance between topics using the cosine similarity. The experimental results on real-world document databases show that the proposed measure finds similar documents more accurately than the cosine similarity owing to reflecting semantic similarity.
URI
http://maxwellsci.com/jp/mspabstract.php?jid=RJASET&doi=rjaset.12.2323http://hdl.handle.net/20.500.11754/31477
ISSN
2040-7467; 2040-7459
DOI
10.19026/rjaset.12.2323
Appears in Collections:
COLLEGE OF ENGINEERING[S](공과대학) > COMPUTER SCIENCE AND ENGINEERING(컴퓨터공학부) > Articles
Files in This Item:
There are no files associated with this item.
Export
RIS (EndNote)
XLS (Excel)
XML


qrcode

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

BROWSE