Twister2 Yarn/Mesos Integration
Twister2 is a loosely coupled component based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments. Twister2 will have a pluggable architecture for deployment on various cluster resource managers including slurm, mesos and yarn. We would like to run Twister2 jobs on Yarn/Mesos as a starting point. a) Choose a framework on top of Yarn/Mesos to interface with. I.e. Aurora, Marathon, Reef b) Write a plugin to Twister2 resource scheduler to start the containers on those systems c) Need to work on the plugin API to improve it along the way. For these purposes we will need four nodes that we can install and experiment on Yarn, Mesos, Aurora, Marathon, Reef, etc. Need to work on the plugin API to improve it along the way.