AWS EMR Jobs: Add Steps to Cluster

{"URL":["/*.*/awa/pa_view_pa_view_ADD_STEPS_TO_CLUSTER_JOB_emr"],"heroDescriptionIdentifier":"ice_hero_EMR_Add_Steps_to_Cluster_Job","customCards":[{"id":"ice_specific_EMR_Add_Steps_to_Cluster_Job","title":"Adding Steps to Cluster Job Parameters","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AWS_EMR/*.*/Agent%20Guide/Content/AWS_EMR/AWS_EMR_AddStepsToCluster_Job.htm","languages":["en-us"]},{"id":"ice_RA_Integration_Report","title":"RA / Integration Reports","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AWS_EMR/*.*/Agent%20Guide/Content/AWS_EMR/AWS_EMR_Jobs_RA_Properties.htm","languages":["en-us"]},{"id":"ice_Add_Step_to_Cluster_Job_script_EMR","title":"Setting Add Steps to Cluster Job Properties through Scripts","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AWS_EMR/*.*/Agent%20Guide/Content/AWS_EMR/AWS_EMR_Script.htm","languages":["en-us"]},{"id":"ice_related_information_EMR_Add_Steps_to_Cluster_Job","title":"Related Information","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AWS_EMR/*.*/Agent%20Guide/Content/AWS_EMR/AWS_EMR_AddStepsToCluster_Job.htm","languages":["en-us"]}]}

Automic Automation AWS EMR Add Steps to Cluster Jobs allow you to schedule and monitor jobs for any of the service instances that exist on the AWS EMR Cockpit.

Defining AWS EMR Add Steps to Cluster Job Properties

On the Add Steps to Cluster Job page you define the parameters relevant to add multiple steps to execute tasks on AWS EMR cluster.

(Click to expand)

Screenshot showing the Add Steps to cluster Job page of the corresponding job.

  • Connection

    AWS EMR Connection object containing the relevant information to connect to the application.

    To search for a Connection object, start typing its name to limit the list of the objects that match your input.

  • Cluster ID

    You can select the Cluster ID on which the steps should run from a picker. For example:

    j-1MYG41H9W79SH

  • Steps

    You can add multiple steps to the AWS cluster in JSON format. Steps are tasks that will run on the cluster. For example:

    [
       {
           "ActionOnFailure": "CONTINUE",
            "HadoopJarStep": {
               "Args": [ "spark-submit", "--deploy-mode","cluster", 
               "s3://wlademoemrsamplebucket/health_violations.py", 
               "--data_source", "s3://wlademoemrsamplebucket/food_establishment_data.csv", 
               "--output_uri" ,"s3://wlademoemrsamplebucketoutput/outputEMR/" ],
               "Jar": "command-runner.jar"
            },
            "Name": "wlademo"
        }
]

    • Note:

    The Report (REP) and the Agent log (PLOG) will show an individual Step ID for each step. These Step IDs are identical to the ones shown on the AWS EMR portal. See also Monitoring AWS EMR Jobs

  • Execution Role ARN

    (Optional) ARN of the AWS EMR cluster that you want to restart.

See also: