MapReduce Scheduling in Cloud Environments

Project Details

Project Lead
Renan DelValle 
Project Manager
Renan DelValle 
Project Members
Renan DelValle, Jessica Hartog, John Weachock  
Supporting Experts
Tak-Lon Wu  
Institution
Binghamton University, Computer Science  
Discipline
Computer Science (401) 

Abstract

This project aims to develop modules for a MapReduce framework capable of efficiently utilizing the resources in a cloud environment. We will conduct research on the problems associated with existing MapReduce implementations and the components and trade-offs necessary for scheduling and migrating tasks dynamically.

Intellectual Merit

This project will advance the state of the art in scheduling for large-scale data intensive applications by taking into account the heterogeneity in the infrastructure of large grids and clouds.

Broader Impacts

The software modules resulting from this project will be released as open-source to the HPC community.

Scale of Use

100-300 nodes, for experiments run a few times a week, for the next 6-12 months.