Boto: how to keep EMR job flow running after completion/failure?
-
18-02-2021 - |
Question
How can I add steps to a waiting Amazon EMR job flow using boto without the job flow terminating once complete?
I've created an interactive job flow on Amazon's Elastic Map Reduce and loaded some tables. When I pass in new steps to the job flow using Boto's emr_conn.add_jobflow_steps(...)
, the job flow terminates after it finishes or fails.
I know I can start a job flow with boto using run_jobflow
with the keep_alive
parameter -- but I'd like to work with flows that are already running.
Solution
If it finishes correctly, it should not terminate with keep_alive=True
. That said, it would normally exit on failure, so you want to add terminate_on_failure="CONTINUE"
to your add_job_steps
arguments.
OTHER TIPS
I use something like this
create with
import boto.emr
conn = boto.emr.connect_to_region('us-west-2')
jobid = conn.run_jobflow(name='cluster-name',
ec2_keyname="yourkeyhere",
num_instances=3,
master_instance_type='m1.medium',
slave_instance_type='m1.medium',
keep_alive=True,
)
add a job to an existing cluster with (wait a little for cluster to be in waiting state)
import boto.emr
conn = boto.emr.connect_to_region('us-west-2')
# get the list of waiting cluster and take the first one
jobid = conn.describe_jobflows(states=["WAITING"])[0].jobflowid
print jobid
flow_steps = list()
runThing = boto.emr.step.ScriptRunnerStep(
name="job step name",
step_args = ["s3://yours3bucket/dosmthg.sh"])
flow_steps.append(runThing)
conn.add_jobflow_steps(jobid, flow_steps)
notes
- you need to have ~/.aws/credentials filled up (aws configure)
- amazon region us-west-2 currently has the more recent ami version
- you can may have to add bootstrap_actions= if you need hive, pig or custom installation steps
You can also do this with the 'KeepJobFlowAliveWhenNoSteps' flag.
response = emr.run_job_flow(
Name="start-my-cluster",
ReleaseLabel="emr-5.3.1",
LogUri='s3://logs',
Instances={
'InstanceGroups': [
{'Name': 'EmrMaster',
'InstanceRole': 'MASTER',
'InstanceType': 'm3.xlarge',
'InstanceCount': 1},
{'Name': 'EmrCore',
'InstanceRole': 'CORE',
'InstanceType': 'm3.xlarge',
'InstanceCount': 2}
],
'Ec2KeyName': 'my-key-name',
'KeepJobFlowAliveWhenNoSteps' : True,
},
Applications=[{'Name': 'Hadoop'}, {'Name': 'Spark'}, {'Name': 'Hive'}],
JobFlowRole='EMR_EC2_DefaultRole',
ServiceRole='EMR_DefaultRole',
VisibleToAllUsers=True,
Steps=[
# steps go here...
]
)