Dynamic Slot Allocation Optimization Framework for HADOOP MapReduce Clusters

Balaji Bodkhe, Sanjay P. Sood

Abstract


Map Reduce is used for large-scale data processing in Big
Data. However, the slot-based MapReduce system (e.g.,
Hadoop MRv1) can suffer from poor performance due to its
unoptimized resource allocation. To solve this problem this
paper identifies and optimizes the resource allocation from
three key aspects. First, due to the pre-configuration of
distinct map slots and reduce slots which are not fungible,
slots can be severely under-utilized. Because map slots might
be fully utilized while reduce slots are empty, and vice-versa.
This paper also proposes an alternative technique called
Dynamic Hadoop Slot Allocation by keeping the slot-based
model. It relaxes the slot allocation constraint to allow slots to
be reallocated to either map or reduce tasks depending on
their needs. Second, the speculative execution can tackle the
straggler problem, which has shown to improve the
performance for a single job but at the expense of the cluster
efficiency. In view of this, we propose Speculative Execution
Performance Balancing to balance the performance between a
single job and a group of jobs. Third, delay scheduling has
shown to improve the data locality but at the cost of fairness.
Additionally, the paper propose a technique called Slot
PreScheduling that can improve the data locality but with no
impact on fairness. Finally, by combining these techniques
together, we form a step-by-step slot allocation system called
DynamicMR that can improve the performance of
MapReduce workloads substantially.The abstract is to be in
fully-justified italicized text as it is here, below the author
information.

Full Text:

PDF

References


F.Ahmad,S.Y.Lee, M. Thottethodi, T. N.

Vijaykumar. PUMA: Purdue MapReduce Benchmarks

Suite. ECE TechnicalReports, 2012.

G. Ananthanarayanan, S. Kandula, A.

Greenberg, I. Stoica, Y. Lu, B. Saha, and

E. Harris, Reining in the outliers in

map-reduce clusters using mantri, in

OSDI’10, pp. 1-16, 2010.

J. Dean and S. Ghemawat. MapReduce: Simplified Data

Processing on Large Clusters, In OSDI’04, pp. 107-113,

A Recommender System Based on a Machine Learning

Algorithm for B2C Portals http://ieeexplore.ieee.org.

A Case-Based Recommendation Approach for Market

Basket Data

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber

=6894473newsearch=truequeryText=Casebased.

Trust-based decision-making for the adaptation of public

displays in changing social contexts

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber

=6596068newsearch=Study.

Trust enabled Argumentation Based Recommender

System http://ieeexplore.ieee.org/articl.

M. Zaharia, A. Konwinski , A.D. Joseph , R. Katz , I.

Stoica, Improving MapReduce performance in

heterogeneous environments. In OSDI’08, pp.29-42,




 

Copyright © IJETT, International Journal on Emerging Trends in Technology