Dynamic slot allocation for mapreduce workload

  • Unique Paper ID: 144281
  • Volume: 3
  • Issue: 10
  • PageNo: 58-61
  • Abstract:
  • MapReduce is a popular parallel Computing paradigm, for large-scale data processing in cluster and data centers. However, the slot utilization can be low, especially when Hadoop Fair scheduler is used, due to the pre-allocation of slots among map and reduce tasks, and the order that map tasks followed by reduce task in a typical MapReduce environment. To address this problem, we propose to allow slots to be dynamically allocated to either map or reduce tasks depending on their actual requirement. Specifically, we have proposed two types of Dynamic Hadoop Fair scheduler (DHFS), for two different levels of fairness (i.e., cluster and pool level).The experimental results show performance significantly while guaranteeing the fairness.
add_icon3email to a friend

Copyright & License

Copyright © 2025 Authors retain the copyright of this article. This article is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

BibTeX

@article{144281,
        author = {M.S.Harini Lakshmi and Fathima nisha S and Manjula D and A. Sheelavathi},
        title = {Dynamic slot allocation for mapreduce workload},
        journal = {International Journal of Innovative Research in Technology},
        year = {},
        volume = {3},
        number = {10},
        pages = {58-61},
        issn = {2349-6002},
        url = {https://ijirt.org/article?manuscript=144281},
        abstract = {MapReduce is a popular parallel Computing paradigm, for large-scale data processing in cluster and data centers. However, the slot utilization can be low, especially when Hadoop Fair scheduler is used, due to the pre-allocation of slots among map and reduce tasks, and the order that map tasks followed by reduce task in a typical MapReduce environment. To address this problem, we propose to allow slots to be dynamically allocated to either map or reduce tasks depending on their actual requirement. Specifically, we have proposed two types of Dynamic Hadoop Fair scheduler (DHFS), for two different levels of fairness (i.e., cluster and pool level).The experimental results show performance significantly while guaranteeing the fairness.},
        keywords = {Map reduces Hardtop, Fair Scheduler, Dynamic Scheduling, Slots allocation},
        month = {},
        }

Cite This Article

  • ISSN: 2349-6002
  • Volume: 3
  • Issue: 10
  • PageNo: 58-61

Dynamic slot allocation for mapreduce workload

Related Articles