An Efficient Replication Method for Map Reduce Scaling

DOI

No DOI

Published In

Volume 3, Issue 4 (2022)

Published Online

July-August 2022

Pages

96-100

Article Type

Research Article

Check for updates

No DOI available for cross-reference

Abstract: Recently, research and utilization of distributed storage and processing systems for large-scale data processing and management are becoming important. Hadoop is widely used as a representative distributed storage and processing framework. Task assignment in Map- Reduce, which is performed based on the Hadoop distributed file system, is assigned as close as possible considering the locality of data. However, in the data analysis work in Map-Reduce, there is data that is frequently requested depending on the type of work. In this case, due to the low locality of the data, it causes problems of an increase in execution time and delay of data transmission. In this paper, we propose a Least Recently Used LRU-data replication technique according to the data access pattern to improve the processing speed of Map-Reduce. In the proposed method, data locality is improved by using a replica optimization algorithm for LRU- data showing high access frequency according to the data access pattern, and consequently the operation time is reduced. As a result of the performance evaluation, it was confirmed that the load of the access frequency was reduced compared to the existing technique.