Workload Characteristic Oriented Scheduler for MapReduce

摘要: Applications in many areas are increasingly developed and ported using the Map Reduce framework (more specifically, Hadoop) to exploit (data) parallelism. The application scope of has been extended beyond original design goal which was large-scale data processing. This extension inherently makes a need for scheduler explicitly take into account characteristics job two main goals efficient resource use performance improvement. In this paper, we study scheduling strategies effectively deal with different workload characteristicsCPU intensive I/O intensive. We present Workload Characteristic Oriented Scheduler (WCO), strives co-locating tasks possibly jobs complementing usage characteristics. WCO is characterized by its essentially dynamic adaptive decisions information obtained from characteristic estimator. primarily estimated sampling help some static task selection strategies, e.g., Java byte code analysis. Results extensive experiments 11 benchmarks 4-node local cluster 51-node Amazon EC2 show 17% improvement on average terms throughput situation co-existing diverse workloads.