A speculative execution strategy based on node classification and hierarchy index mechanism for heterogeneous Hadoop systems
作者： Qi Liu , Weidong Cai , Jian Shen , Zhangjie Fu , Xiaodong Liu
DOI: 10.23919/ICACT.2017.7890240
关键词:
摘要: MapReduce (MR) has been widely used to process distributed large data sets. MRV2 working on Yarn, as a more advanced programing model, gained lots of concerns. Meanwhile, speculative execution is known an approach for dealing with same problems by backing up those tasks running low performance machine higher one. In this paper, we have modified some pitfalls and taken heterogeneous environment into consideration. Besides, Node classification novel hierarchy index mechanism created. We also implemented it in Hadoop-2.6 the strategy above called Speculation-NC while optimized Hadoop Hadoop-NC. Experiment results show that our method can correctly backup task, improve decrease time resource consumption compared traditional strategies.
sci-hub.se 参考文章(11)
Matei Zaharia, Andy Konwinski, Anthony D. Joseph, Ion Stoica, Randy Katz, Improving MapReduce performance in heterogeneous environments operating systems design and implementation. pp. 29- 42 ,(2008) , 10.5555/1855741.1855744
Zhangjie FU, Xingming SUN, Qi LIU, Lu ZHOU, Jiangang SHU, Achieving Efficient Cloud Search Services: Multi-Keyword Ranked Search over Encrypted Cloud Data Supporting Parallel Computing IEICE Transactions on Communications. ,vol. 98, pp. 190- 200 ,(2015) , 10.1587/TRANSCOM.E98.B.190
Yanfeng Zhang, Qixin Gao, Lixin Gao, Cuirong Wang, PrIter: A Distributed Framework for Prioritizing Iterative Computations IEEE Transactions on Parallel and Distributed Systems. ,vol. 24, pp. 1884- 1893 ,(2013) , 10.1109/TPDS.2012.272
Qi Chen, Cheng Liu, Zhen Xiao, Improving MapReduce Performance Using Smart Speculative Execution Strategy IEEE Transactions on Computers. ,vol. 63, pp. 954- 967 ,(2014) , 10.1109/TC.2013.15
Jing Tai Piao, Jun Yan, Computing resource prediction for mapreduce applications using decision tree asia-pacific web conference. pp. 570- 577 ,(2012) , 10.1007/978-3-642-29253-8_51
Michael Armbrust, Armando Fox, Rean Griffith, Anthony D Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, Matei Zaharia, None, A view of cloud computing Communications of The ACM. ,vol. 53, pp. 50- 58 ,(2010) , 10.1145/1721654.1721672
Yunhong Gu, Robert L. Grossman, Sector and Sphere: the design and implementation of a high-performance data cloud Philosophical Transactions of the Royal Society A. ,vol. 367, pp. 2429- 2445 ,(2009) , 10.1098/RSTA.2009.0053
Wenbin Fang, Bingsheng He, Qiong Luo, Naga K. Govindaraju, Mars: Accelerating MapReduce with Graphics Processors IEEE Transactions on Parallel and Distributed Systems. ,vol. 22, pp. 608- 620 ,(2011) , 10.1109/TPDS.2010.158
Andréa Matsunaga, José A.B. Fortes, On the Use of Machine Learning to Predict the Time and Resources Consumed by Applications 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. pp. 495- 504 ,(2010) , 10.1109/CCGRID.2010.98
Xiaolong Wen, Genqiang Gu, Qingchun Li, Yun Gao, Xuejie Zhang, None, Comparison of open-source cloud management platforms: OpenStack and OpenNebula fuzzy systems and knowledge discovery. pp. 2457- 2461 ,(2012) , 10.1109/FSKD.2012.6234218