192 0

An Efficient Key Partitioning Scheme for Heterogeneous MapReduce Clusters

Title
An Efficient Key Partitioning Scheme for Heterogeneous MapReduce Clusters
Author
이춘화
Keywords
Cloud Computing; Context-awareness; Hadoop; Heterogeneous system; MapReduce
Issue Date
2016-01
Publisher
Global IT Research Institute
Citation
International Conference on Advanced Communication Technology, v. 2016-March, Article number 7423394, Page. 364-367
Abstract
Hadoop is a standard implementation of MapReduce framework for running data-intensive applications on the clusters of commodity servers. By thoroughly studying the framework we find out that the shuffle phase, all-to-all input data fetching phase in reduce task significantly affect the application performance. There is a problem of variance in both the intermediate key's frequencies and their distribution among data nodes throughout the cluster in Hadoop's MapReduce system. This variance in system causes network overhead which leads to unfairness on the reduce input among different data nodes in the cluster. Because of the above problem, applications experience performance degradation due to shuffle phase of MapReduce applications. We develop a new novel algorithm; unlike previous systems our algorithm considers a node's capabilities as heuristics to decide a better available trade-off for the locality and fairness in the system. By comparing with the default Hadoop's partitioning algorithm and Leen algorithm, on the average our approach achieve performance gain of 29% and 17%, respectively. © 2016 Global IT Research Institute (GiRI).
URI
http://ieeexplore.ieee.org/document/7423394/http://hdl.handle.net/20.500.11754/30330
ISBN
978-8-9968-6506-3
ISSN
1738-9445
DOI
10.1109/ICACT.2016.7423394
Appears in Collections:
COLLEGE OF ENGINEERING[S](공과대학) > COMPUTER SCIENCE AND ENGINEERING(컴퓨터공학부) > Articles
Files in This Item:
There are no files associated with this item.
Export
RIS (EndNote)
XLS (Excel)
XML


qrcode

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

BROWSE