Twister2 Yarn/Mesos Integration

Project ID
FG-535
Project Categories
Computer Science
Abstract

Twister2 is a loosely coupled component based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments. Twister2 will have a pluggable architecture for deployment on various cluster resource managers including slurm, mesos and yarn. We would like to run Twister2 jobs on Yarn/Mesos as a starting point. a) Choose a framework on top of Yarn/Mesos to interface with. I.e. Aurora, Marathon, Reef b) Write a plugin to Twister2 resource scheduler to start the containers on those systems c) Need to work on the plugin API to improve it along the way. For these purposes we will need four nodes that we can install and experiment on Yarn, Mesos, Aurora, Marathon, Reef, etc. Need to work on the plugin API to improve it along the way.

Use of FutureSystems
We will run the developed components in this cluster environment and conduct performance tests. We will be installing YARN and MESOS, and running Twister2 on them. We plan to use around 4 nodes. We may also install Aurora, Marathon, and REEF.
Scale of Use
We expect this project to take months. We will be experimenting with these systems maybe daily during this time.