Reducing Data Skew with Round Robin Horizontal Partitioning of Data for Distributed Association Rule Mining of Large Data Set

Abstract

High growth in data size is observed due use of
computer in all field. This data is not useful for decision making in
business, unless is mined to extract interesting knowledge from it.
For analyzing such data and extracting true knowledge from it,
various data mining techniques are used. Association rule mining
is one of them; it aims at finding associations or relations among
data. As size of the data increase, knowledge discovery on this high
volume data becomes slow, with conventional data mining
technique, as it has to be done serially. The number of data records
may make the learning process very slow. The solution to the
problem is to speed-up the learning process with the help of parallel
or distributed techniques. Through mining, interesting relations and
patterns between variables of large database can be observed using
the distributed mining algorithms. The performance in terms of time
complexity data mining algorithm can be from O(N) to lower bound
O(N/k) with parallel or distributed approach, where N = number of
data instances and k = number of nodes in distributed system[1].
Partitioning and distribution of data on different nodes in distributed
system may lead to data skew and intern a problem in computing
support and confidence. This paper addresses the distributed
association rule mining on large datasets and merging rules in single
rule set. This system horizontally distributes large data set using
round robin method and association rule mining using Apriori
algorithm is performed with global support count at least s and
confidence count at least c. Duplicate rules in the system create rule
redundancy. Duplicate rules are found and redundancy is removed
from rule set before final merger of the rules at central server. Data
security issue in distributed mining has been handled by many
researchers so it is not addressed here. The speed up is acquired
with proposed method is significant along with utilization of
available computing resources.