Hadoop built on Apache Mesos layer

Project Details

Project Lead
John Lockman  
Project Manager
John Lockman  
Project Members
Ritu Arora  
University of Texas at Austin, Texas Advanced Computing Center  
Computer Science (401) 


This project allocation is being requested for evaluating “Hadoop built on Apache Mesos layer” in the FutureGrid environment. The Apache Mesos layer is a common resource sharing layer over which diverse frameworks like Hadoop and Spark can run at the same time. For testing the installation, we are particularly interested in running I/O intensive and memory-intensive applications/benchmarks using the MapReduce algorithm. The lessons learnt during this project will guide our strategy for comparative study of Hadoop and Apache Mesos on different high performance computing platforms having varying architecture and performance characteristics.

Intellectual Merit

This project will attempt to demonstrate the usability of Hadoop running in the Apache Mesos environment and provide a method for researchers to run large mapreduce problems in the FutureGrid

Broader Impacts

Once proven stable, Apache Mesos can be used by researchers to run large Hadoop problems in the FutureGrid and other XSEDE resources.

Scale of Use

Typical use will be a few VMs for an experiment. We may also need to gather a handful of nodes for a few days at a time.