FG Twister Appliance Tutorial

I. Introduction

This tutorial will demonstrate how to setup a private cluster for a Twister appliance on FutureGrid.

II. Prerequisites

1. Setting up GroupVPN for Creating a Virtual Cluster

This is the process of creating a virtual private network (VPN) interconnecting your virtual machine instances in order to form a virtual cluster. GroupVPN is a group secured VPN network using the IPOP IP-over-P2P virtual network to create virtual clusters for running parallel and batch applications. GroupVPN can be used to easily consolidate physical machines, virtual machine instances, and cloud instances into a single virtual cluster.

Click here to create an account at www.grid-appliance.org.

2. Set up a GroupVPN

Log into www.grid-appliance.org. Follow the instructions at GroupVPN Web User Interface and configure a GroupVPN with the desired configuration.

3. Set up a Group Appliance

The grid appliance website allows for the creation of floppy disks with the configuration files required by GroupVPN. To do this, follow the instruction here.

4. Choose the Twister Grid Appliance Image

Choose a gridappliance image: a Twister grid appliance image (emi-F0B8194D) has been tailored for FutureGrid. Deploy as many nodes preferable in private cluster. For information on how to access Eucalyptus features, follow the tutorial.

III. Deploying a grid appliance connected to your own pool using GroupVPN

To connect to any other GroupVPN pool, place a floppy disk containing the GroupVPN configuration files inside the appliance in the /opt/grid_appliance/etc directory and restart the grid appliance service. This floppy can be generated using the grid appliance.org web front-end. In the case of executing a private cluster for a Twister appliance, download the server and worker floppies. Designate one node as the server and the remaining nodes as workers. The steps for starting and stopping the grid appliance service are as outlined below:

  1. Stop the grid_appliance service in the grid-appliance

  2. Download the GroupVPN floppy to your local system from the grid-appliance website. Transfer it to the Grid Appliance workspace using scp or sftp.
    scp - i userkey.private floopy.img (server) root@public.ip.address:/opt/grid_appliance/etc/
    scp - i userkey.private floopy.img (worker) root@public.ip.address:/opt/grid_appliance/etc/
  3. SSH back into grid appliance and start the grid appliance service

    ssh -i userkey.private root@public.ip.address /etc/init.d/grid_appliance.sh start

IV. Setup Twister Environment

1. Edit $TWISTER_HOME/bin/twister.properties and set the correct paths to the following:

nodes_file = /opt/Twister/bin/nodes

app_dir = /opt/Twister/apps

data_dir = /opt/Twister/data

pubsub_broker = Naradabrokering

2. Edit $TWISTER_HOME/bin/nb.properties and set broker_host = (to the IP address on the tapipop interface of the machine you designated as the server)

3. Edit $TWISTER_HOME/bin/nodes and add your IP addresses on the tapipop interface of the machines you designated as workers.

V. Twister-Kmeans

Open two terminals - One for starting the broker and the second for starting Twister.

For steps 2 - 4, execute the following commands on the second terminal.

1. Start Twister and Naradabrokering

In Terminal 1:

$ cd /opt/Naradabrokering/bin/
$ chmod 755 startbr.sh
$ ./startbr.sh

In Terminal 2: 

$ cd /opt/Twister/bin
$ chmod 755 start_twister.sh
$ ./start_twister.sh

2. Ant  Twister-Kmeans

$ cd /opt/Twister/samples/kmeans
$ ant
$ cd ../../lib
$ mv Twister-Kmeans-0.9.jar ../apps/

3. Create Twister-Kmeans Data Folder

$ cd /opt/Twister/bin
$ ./twister.sh mkdir kmeans

4. Generate Twister-Kmeans Data

$ cd /opt/Twister/samples/kmeans/bin/
$./gen_data.sh init_clusters.txt 2 3 kmeans km_data 3 30000

$ ./create_partition_file.sh kmeans km ../samples/kmeans/bin/p.pf

Execute Twister-Kmeans application

$ ./run_kmeans.sh init_clusters.txt 3 p.pf

The output is as follows:

A video detailing the above steps can be shown here password: tutorial
Your rating: None