Error: Jobflow entered COMPLETED while waiting to ssh

Question

I just started to practice AWS EMR.

I have a sample word-count application set-up, run and completed from the web interface.

Following the guideline here, I have setup the command-line interface.

so when I run the command:

./elastic-mapreduce --list

I receive

j-27PI699U14QHH     COMPLETED      ec2-54-200-169-112.us-west-2.compute.amazonaws.comWord count
   COMPLETED      Setup hadoop debugging        
   COMPLETED      Word count        

Now, I want to see the log files. I run the command

./elastic-mapreduce --ssh --jobflow j-27PI699U14QHH

Then I receive the following error:

Error: Jobflow entered COMPLETED while waiting to ssh

Can someone please help me understand what's going on here?

Thanks,

Answer 1

When you setup a job on EMR, this means that Amazon is going to provision a cluster on-demand for you for a limited amount of time. During that time, you are free to ssh to your cluster and look at the logs as much as you want, but by the time your job has finished running, then your cluster is going to be taken down ! At that point, you won't be able to ssh anymore because your cluster simply won't exist.

The workflow typically looks like this:

• Create your jobflow
• It will be for a few minutes in status STARTING. At that point if you try to run ./elastic-mapreduce --ssh --jobflow <jobid> it will simply wait because the cluster is not available yet.
• After a while the status will switch to RUNNING. If you had already started the ssh command above it should automatically connect you to your cluster. Otherwise you can initiate your ssh command now and it should connect you directly without any wait.
• Depending on the nature of your job, the RUNNING step could take a while or be very short, it depends what amount of data you're processing and the nature of your computations.
• Once all your data has been processed, the status will switch to SHUTTING_DOWN. At that point, if you already sshed before you will get disconnected. If you try to use the ssh command at that point, it will not connect.
• Once the cluster has finished shutting down it will enter a terminal state of either COMPLETED or FAILED depending on whether your job succeeded or not. At that point your cluster is no longer available, and if you try to ssh you will get the error you are seeing.

Of course there are exceptions, you could setup an EMR cluster in interactive mode, for example you just want to have Hive setup and then ssh there and run Hive queries and you would have to take your cluster down manually. But if you just want a MapReduce job to run, then you will only be able to ssh for the duration of the job.

That being said, if all you want to do is debugging, there is not even a need to ssh in the first place ! When you create your jobflow, you have the option to enable debugging, so you could do something like that:

./elastic-mapreduce --create --enable-debugging --log-uri s3://myawsbucket

What that means is that all the logs for your job will end up being written to the S3 bucket specified (you have to own this bucket of course and have permission to write to it). Also if you do that, you can go into the AWS console afterwards in the EMR section, and you will be able to see next to your job a button to debug as shown below in the screenshot, this should make your life much easier: