How to properly configure the Amazon EC2 AMI for 'hadoop-ec2'?

Question

I am trying to launch an instance on Amazon EC2. I have researched this problem extensively, but I have not found any helpful information.

When I run the command hadoop-ec2 launch-cluster mycluster 2, I receive the following error message:

Starting master with AMI.
Required parameter 'AMI' missing (-h for usage)

I have entered my AWS key, AWS secret key, AWS key pairs, etc. I am using hadoop-1.0.4. I am using the default S3 bucket (hadoop-images), but I have tried many other AMIs and I always get the same error message.

Has anybody experience this problem before?

Answer 1

The basic issue is that the search for images the launch-hadoop-master script performs is not returning any results. The most likely cause of this due to the different AMIs that are available in different regions (but it could be due to any changes you've made to S3_BUCKET and HADOOP_VERSION in hadoop-ec2-env.sh).

From the launch-hadoop-master script:

# Finding Hadoop image
AMI_IMAGE=`ec2-describe-images -a | grep $S3_BUCKET 
                                  | grep $HADOOP_VERSION
                                  | grep $ARCH
                                  | grep available
                                  | awk '{print $2}'`

# Start a master
echo "Starting master with AMI $AMI_IMAGE"

So, it appears that AMI_IMAGE is not being set to a valid image and thus the search for AMIs that match the various grep filters is failing (the defaults for the Hadoop 1.0.4 distribution are S3_BUCKET is hadoop-images, HADOOP_VERSION is 0.19.0 and ARCH is x86 if you're using m1.small instances). If you search the public AMIs in the US-West-2 region, you'll see that there aren't many Hadoop images, but if you search the public AMIs in the US-East-1 region, you'll see that there are quite a few. Thus, one way around this issue is to work in the US-East-1 region (this is simplest) or, alternatively, set EC2_URL in your login script via export EC2_URL=https://ec2.us-east-1.amazonaws.com but now you need make sure you put your keys in this region from the AWS console.

If you did indeed change HADOOP_VERSION to 1.0.4, I'll note that

ec2-describe-images -a | grep hadoop-images 
                       | grep "1.0.4"'
                       | grep x86
                       | grep available 

doesn't return any images in the US-East-1 region. Note that the version (HADOOP_VERSION) of the Hadoop distribution that you are running the hadoop-ec2 command from does not need to be the same as the version of Hadoop that the images will be running.

Lastly, as a blunt fix, you could find the AMI that you want to use, and force set AMI_IMAGE to the image name in the launch-hadoop-master and launch-hadoop-cluster scripts.