Federating HPC, Cyberinfrastructure and Clouds using CometCloud

Project ID
FG-288
Project Categories
Computer Science
Project Keywords
Completed
Abstract
Clouds are rapidly joining high-performance Grids as viable computational platforms for scientific exploration and discovery, and it is clear that production computational infrastructures will integrate both paradigms in the near future. As a result, understanding usage modes that are meaningful in such a hybrid infrastructure is critical. For example, there are interesting application workflows that can benefit from such hybrid usage modes to, perhaps, reduce times to solutions, reduce costs (in terms of currency or resource allocation), or handle unexpectedruntime situations (e.g. unexpected delays in scheduling queues or unexpected failures). The primary goal of this project is to use CometCloud to create a federation that integrate resources from different infrastructures namely FutureGrid, XSEDE, Grid’5000, Rutgers Discovery Informatics Institute resources and, possibly, resources from other collaborating institutions. Moreover, the infrastructure will enable scale out to public clouds such as Amazon EC2 on-demand. In this way, we will be able to expose the different cyber-infrastructure ecosystems to scientific and engineering applications as Cloud services. Additionally, scientifics will be able to create their own applications on top of CometCloud using different programming models like master/worker, workflows or map/reduce.
Use of FutureSystems
We intend to include different FutureGrid sites into the Federation to be able to scale application across sites. FutureGrid cloud services would be used as part of the infrastructure or to scale out the infrastructure on demand though the CometCloud cloudburst capabilities.
Scale of Use
We will need a few VMs to setup the environment and prepare the actual experiments. It may take some weeks.
Then, we will run a set of run at different scales using different systems and different configurations. Analysis will be performed between experiments in order to evaluate only key scenarios of interest. This process may take some months but the use of the resources will not be continued.