Hadoop/MapReduce Lab

Setting up a cluster manually on AWS is doable, but involves a lot of manual clicking and setting things up. We're instead going to create Hadoop clusters at AWS using a program called StarCluster, which is a set of Python scripts designed by some folks at MIT to make this more automatic.

StarCluster has already been involved on the department lab computers. If you want to use it on your own computer, you'll need to install it yourself.

The below instructions are based heavily on this StarCluster quick-start guide. Feel free to refer to it instead if you want to see different wording on the same instructions I have below.

To log in to your cluster, issue this command:

$ starcluster sshmaster mycluster

The sshmaster option indicates that you are connecting to the master node of the cluster, and it's the machine from which you'll launch your Hadoop jobs on the cluster. If this works, you may need to answer "yes" to a question about an RSA key fingerprint, and then you should see a welcome screen with some ASCII art for StarCluster, and find yourself at a UNIX prompt.

If this login does not work, ask for help.

Type exit (or ctrl-D) to go back to your local machine. Make sure you are in the directory where your Trivial.java file is. The version of Hadoop running on the Amazon cluster is not the same one running in our labs, so you'll need to transfer the Java file over to the cluster, and compile it again over there. Issue the commands below:

$ starcluster put mycluster Trivial.java /root

While you're at it, also transfer over the directory of data that you've been working with:

$ starcluster put mycluster trivialInput /root

SSH into your cluster again:

$ starcluster sshmaster mycluster

Once you have connected, issue an ls command to verify that your files are there on the cluster as expected.

As before, you'll need to compile your Hadoop program. Hopefully it has no errors; it's the same program you ran locally. We are using a different version of Hadoop on the cluster than in the labs, but the differences from a programming perspective are exceedingly small, and shouldn't affect anything we do. Since the versions of Hadoop are different, transferring the jar file that we created locally won't work.

You'll again need to set up some configuration settings on the cluster for the locations of Java and Hadoop. As before, you'll need to add the following three lines to a file. On the cluster, that file is .bashrc (not .bashrc_profile as it is on the department macs). Since we're in a command-line situation, you'll need to use an editor that works in a terminal window. This is a great occasion to learn how to use vim or emacs! All of that said, there is a less capable editor known as pico that's pretty easy to use if you're just looking to make a quick edit. Issue the command

root@master:~# pico .bashrc

and then scroll down with the cursor keys to the bottom of the file. Copy and paste in the following two lines below:

export JAVA_HOME=/usr/lib/jvm/default-java
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

See if you can figure out how to save the file from there.

Once you have saved the above changes, log out of the cluster (type exit), then ssh back in again. If all worked out, the following compile commands should now work:

root@master:~# hadoop com.sun.tools.javac.Main Trivial.java

If it works, then create a jar file again as you did earlier in this lab.

root@master:~# jar cf trivial.jar Trivial*.class

You'll next need to transfer any input files onto HDFS, which is the cluster file system. You've already transferred trivialInput from your department computer to the cluster, but that directory is currently sitting in your root home directory on the master computer. That is simply sitting on the normal local hard disk for that master computer; it is not the same thing as putting the file into HDFS, which is the distributed file system with replication that Hadoop uses. (Hadoop automatically distributes your data over the multiple computers in your cluster as it best sees fit, and it also replicates it over multiple locations.)

You can access the cluster filesystem using the command hadoop fs. This has a bunch of UNIX-like options you can use with it. (Don't type the # with any of the commands below, it's there to indicate that you're typing at a terminal prompt on the cluster.)

# hadoop fs -help

Put the input data directory into HFDS using -put. In the below, don't forget to include that / at the end of trivialInput. Without it, it will copy in the files in trivialInput, but not within that directory.

(below modified on 2/29)

# hadoop fs -put trivialInput /user/root

Now you should be able to see the input file in HDFS:

# hadoop fs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2016-02-26 11:09 /user/root/trivialInput
root@master:~# hadoop fs -ls trivialInput
Found 2 items
-rw-r--r--   3 root supergroup     183773 2016-02-26 11:09 /user/root/trivialInput/pg19778.txt
-rw-r--r--   3 root supergroup     178845 2016-02-26 11:09 /user/root/trivialInput/pg2265.txt

Note that you DO NOT need to copy trivial.jar into HDFS. It should stay in your home directory on the cluster.

We now have all the pieces in place to run a job on the cluster! First, though, there are some nifty web-based tools you should take a look at.

You'll first need to find the public DNS name for your cluster master computer. If you are still logged into the cluster, type exit (or ctrl-D) so you are back at your lab machine again. Then type:

starcluster listclusters

This will show you some information for your cluster. For me, it looks like:

-----------------------------------------
mycluster (security group: @sc-mycluster)
-----------------------------------------
Launch time: 2016-02-26 04:57:57
Uptime: 0 days, 00:51:58
Zone: us-east-1c
Keypair: mykey
EBS volumes: N/A
Cluster nodes:
     master running i-86608d05 ec2-54-163-131-240.compute-1.amazonaws.com
    node001 running i-85608d06 ec2-174-129-164-12.compute-1.amazonaws.com
    node002 running i-84608d07 ec2-54-145-159-21.compute-1.amazonaws.com
    node003 running i-9b608d18 ec2-54-204-211-159.compute-1.amazonaws.com
Total nodes: 4

You want to find the row associated with the master node. The rightmost string there (for me, ec2-54-163-131-240.compute-1.amazonaws.com) is the public DNS that you can use to construct a URL for monitoring the status of your Hadoop cluster in a web browser. You can copy and paste that DNS name into a browser window, and add a port number. Different port numbers get you different information:

• Port 50030 is the JobTracker interface for the cluster. It displays information on all of the jobs that are currently running.
• Port 50070 displays information about the cluster's distributed filesystem. This is probably of less immediate interest to you, but still interesting.

So for me, based on the above output from StarCluster, I would visit the following two URLs:

http://ec2-54-163-131-240.compute-1.amazonaws.com:50030
http://ec2-54-163-131-240.compute-1.amazonaws.com:50070

Make sure you can view those web pages, and ask for help if you need it.

Now, while you've got the first web page above visible (the 50030 one), go back to your terminal window, ssh to your cluster again, and start up your job just as you did on the lab machine. Note that the input and output paths now represent folders on HDFS.

$ starcluster sshmaster mycluster
# hadoop jar trivial.jar Trivial trivialInput output

Again, you should get a bunch of output, hopefully ending with reports of success. If you keep refreshing the job tracker page, you should be able to see your job as either in progress or complete. Click on the job ID and explore the information available.

(Note that if you already have an output directory in HDFS, possibly from a previous run, the above hadoop command will give you an error indicating as such. You'll need to remove that output directory, or move it somewhere else. Use hadoop fs -help to look for commands to remove or move directories.)

Congrats! You've officially executed a MapReduce job on a real Hadoop cluster!

If you want to look at the job output back on your own lab computer, you'll first need to get it out of HDFS, then transfer it back to the lab:

# hadoop fs -get output/part-r-00000 .
# ls
# exit
$ starcluster get mycluster part-r-00000 .
$ head part-r-00000

Amazon charges money for these clusters. How much? Amazon's pricing model is a bit confusing, but here's what I know. Unless you've changed the type of node in your config file, it will create each node of type m1.small. Here is the webpage with pricing for so-called "previous generation" instances, which includes m1.small. (We're using previous generation instances instead of current onces because StarCluster doesn't play nicely (yet) with the new ones. The prices fluctuate all the time. At the time I'm writing this lab, the node type m1.small is selling for $0.044 per hour. If you started four of them, that means you're paying approximately sixteen cents per hour. You can do the math yourself on how much it will then cost you based on how much you leave it running. Remember that if you successfully got your AWS Educate credits, that gets you $100 that you can use.

(Someone will notice that Amazon offers an "AWS Free Tier" with t2.micro instances. Sadly, t2.micro doesn't seem to play nicely with StarCluster. If you want to go this route, you'll have to manually configure the cluster and Hadoop via Amazon's web interface instead of with StarCluster.)

StarCluster also creates an EBS (Elastic Block Storage) 8GB disk associated with each volume. Pricing on that is here. Again, prices change all the time for this. StarCluster creates so-called standard magnetic volumes, which at the moment I'm writing this lab that costs $0.05 per GB-month. In other words, you pay $0.05 for each GB that survives a month. Again, you can do the math to figure out what that means for your cluster, but this amounts to a small amount of money that should easily be covered by your AWS Educate credits.

You should regularly check your Amazon billing status. Amazon can show you how much charges you have incurred. This does not update in real-time, but often can be delayed by a few hours or up to a day. Until your clusters are shut down for good, you should be regularly checking this to make sure that you are not being charged more than you expect. To check your billing status, visit aws.amazon.com, and near the top right of the browser window, select "My Account," and then "Billing & Cost Management." You'll see your current charges there, but this number is very deceiving; at least sometimes for me, it is the amount you will be paying out of pocket, and often appears to be $0 so long as your AWS Educate credits are covering it. That's fine, but this doesn't help you know if you're blowing quickly through your AWS Educate money. The place I've found that's accurate is if you click the "Bill Details" button at the top right of the billing screen, and then click on the "Expand All" that appears below the summary. If you scroll down, you'll then see your actual charges as well as what is being credited. Again, this is not always refreshed right away, so come back often.

Don't miss this section! Leaving your cluster running forever will eventually cost you money!

At this point, you've got a choice: you can leave your cluster running for the next week or so as you'll be working on assignments, or you can terminate the cluster and create a new one later. If you terminate it and make a new one later, you won't need to update your .starcluster/config (that's done), but you would need to update your .bashrc on the cluster again once you've made a new one. If you leave your cluster running, Amazon will continue to charge you money for it. How to proceed is up to you.

When you are ready to shut down your cluster for good, issue this command, which will take bit of time to run:

$ starcluster terminate mycluster

Answer y when it asks if you want to terminate the cluster.

To make sure everything is really shut down, you might issue the following commands after termination is complete:

$ starcluster listclusters
$ starcluster listinstances
$ starcluster listvolumes

Make sure to continue to check your AWS billing for the next few days after you've terminated your last cluster. If you accidentally left something running, you'll continue to get billed.

Starting up clusters again is pretty easy, so I've been choosing to shutdown my clusters when done with a particular session, and start them up again later. If I'm running a long Hadoop job, of course, I need to keep it running.