Document Similarity Measure Based on the Earth Mover's Distance Utilizing Latent Dirichlet Allocation
- Title
- Document Similarity Measure Based on the Earth Mover's Distance Utilizing Latent Dirichlet Allocation
- Author
- 김상욱
- Keywords
- Cosine similairty; document similarity; earth mover’s distance; latent dirichlet allocation; semantic similarity
- Issue Date
- 2016-01
- Publisher
- Medwell
- Citation
- Research Journal of Applied Sciences, Engineering & Technology, v. 12, NO 2, Page. 214-222
- Abstract
- Document similarity is used to search for such documents similar to a query document given. Text-based document similarity is computed by comparing the words in documents. The cosine similarity is the most popular text-based document similarity measure and computes the similarity of two documents based on their common word frequencies. It counts the exactly same words only, so cannot reflect semantic similarity between similar words having the same meaning. We propose a new document similarity measure to solve this problem by using the Earth Mover’s Distance (EMD). The EMD enables to compute the semantic similarity of documents. To apply the EMD to the similarity measure, we need to solve the high computational complexity and to define the distance between attributes. The high computational complexity comes from the large number of words in documents. Thus, we extract the topics from documents by using Latent Dirichlet Allocation (LDA), a document generating model. Since the number of topics is much smaller than that of words, the LDA helps reduce the computational complexity. We define the distance between topics using the cosine similarity. The experimental results on real-world document databases show that the proposed measure finds similar documents more accurately than the cosine similarity owing to reflecting semantic similarity.
- URI
- http://maxwellsci.com/jp/mspabstract.php?jid=RJASET&doi=rjaset.12.2323http://hdl.handle.net/20.500.11754/31477
- ISSN
- 2040-7467; 2040-7459
- DOI
- 10.19026/rjaset.12.2323
- Appears in Collections:
- COLLEGE OF ENGINEERING[S](공과대학) > COMPUTER SCIENCE AND ENGINEERING(컴퓨터공학부) > Articles
- Files in This Item:
There are no files associated with this item.
- Export
- RIS (EndNote)
- XLS (Excel)
- XML