Boto: how to keep EMR job flow running after completion/failure?

Question

How can I add steps to a waiting Amazon EMR job flow using boto without the job flow terminating once complete?

I've created an interactive job flow on Amazon's Elastic Map Reduce and loaded some tables. When I pass in new steps to the job flow using Boto's emr_conn.add_jobflow_steps(...), the job flow terminates after it finishes or fails.

I know I can start a job flow with boto using run_jobflow with the keep_alive parameter -- but I'd like to work with flows that are already running.

Answer 1

If it finishes correctly, it should not terminate with keep_alive=True. That said, it would normally exit on failure, so you want to add terminate_on_failure="CONTINUE" to your add_job_steps arguments.

Answer 2

I use something like this

create with

import boto.emr

conn = boto.emr.connect_to_region('us-west-2')
jobid = conn.run_jobflow(name='cluster-name',
                     ec2_keyname="yourkeyhere",
                     num_instances=3,
                     master_instance_type='m1.medium',
                     slave_instance_type='m1.medium',
                     keep_alive=True,
)

add a job to an existing cluster with (wait a little for cluster to be in waiting state)

import boto.emr

conn = boto.emr.connect_to_region('us-west-2')
# get the list of waiting cluster and take the first one
jobid =  conn.describe_jobflows(states=["WAITING"])[0].jobflowid
print jobid
flow_steps = list()
runThing = boto.emr.step.ScriptRunnerStep(
                        name="job step name",
                        step_args = ["s3://yours3bucket/dosmthg.sh"])
flow_steps.append(runThing)
conn.add_jobflow_steps(jobid, flow_steps)

notes

• you need to have ~/.aws/credentials filled up (aws configure)
• amazon region us-west-2 currently has the more recent ami version
• you can may have to add bootstrap_actions= if you need hive, pig or custom installation steps

Answer 3

You can also do this with the 'KeepJobFlowAliveWhenNoSteps' flag.

response = emr.run_job_flow(
        Name="start-my-cluster",
        ReleaseLabel="emr-5.3.1",
        LogUri='s3://logs',
        Instances={
            'InstanceGroups': [
                {'Name': 'EmrMaster',
                 'InstanceRole': 'MASTER',
                 'InstanceType': 'm3.xlarge',
                 'InstanceCount': 1},
                {'Name': 'EmrCore',
                 'InstanceRole': 'CORE',
                 'InstanceType': 'm3.xlarge',
                 'InstanceCount': 2}
            ],
            'Ec2KeyName': 'my-key-name',
            'KeepJobFlowAliveWhenNoSteps' : True,
        },
        Applications=[{'Name': 'Hadoop'}, {'Name': 'Spark'}, {'Name': 'Hive'}],
        JobFlowRole='EMR_EC2_DefaultRole',
        ServiceRole='EMR_DefaultRole',
        VisibleToAllUsers=True,
        Steps=[
            # steps go here...
        ]
        )