Dynamic Slot Allocation Optimization Framework for HADOOP MapReduce Clusters

Abstract

Map Reduce is used for large-scale data processing in Big
Data. However, the slot-based MapReduce system (e.g.,
Hadoop MRv1) can suffer from poor performance due to its
unoptimized resource allocation. To solve this problem this
paper identifies and optimizes the resource allocation from
three key aspects. First, due to the pre-configuration of
distinct map slots and reduce slots which are not fungible,
slots can be severely under-utilized. Because map slots might
be fully utilized while reduce slots are empty, and vice-versa.
This paper also proposes an alternative technique called
Dynamic Hadoop Slot Allocation by keeping the slot-based
model. It relaxes the slot allocation constraint to allow slots to
be reallocated to either map or reduce tasks depending on
their needs. Second, the speculative execution can tackle the
straggler problem, which has shown to improve the
performance for a single job but at the expense of the cluster
efficiency. In view of this, we propose Speculative Execution
Performance Balancing to balance the performance between a
single job and a group of jobs. Third, delay scheduling has
shown to improve the data locality but at the cost of fairness.
Additionally, the paper propose a technique called Slot
PreScheduling that can improve the data locality but with no
impact on fairness. Finally, by combining these techniques
together, we form a step-by-step slot allocation system called
DynamicMR that can improve the performance of
MapReduce workloads substantially.The abstract is to be in
fully-justified italicized text as it is here, below the author
information.