Hadoop Testing

Project Details

Project Lead
Bryon Gill 
Project Manager
Bryon Gill 
Supporting Experts
Shava Smallen, Zhenhua Guo  
Institution
Carnegie Mellon University, Pittsburgh Supercomputing Center  
Discipline
Computer Science (401) 

Abstract

Comparison testing and experimentation for cluster-based Hadoop for comparison with performance on PSC at Blacklight.

Intellectual Merit

This project will enhance our understanding of Hadoop when used on a large shared-memory machine as compared to a traditional cluster installation, delineating use cases where each configuration will be most likely to yield the best results.

Broader Impacts

With a large shared-memory Hadoop installation, scientific research that involves iterative processing of large data sets should see significant gains.

Scale of Use

I will initially want to provision a cluster of 8 nodes, I will likely want to expand that cluster at some point in the future to match the scale of my experiments on Blacklight.