Google Cloud Dataproc Jobs: Setting Job Properties Through Scripts

Attributes defined for an object affect how the object is processed. Script statements allow you to dynamically change those object attributes and assign new values when the task is generated. The new values are only valid for the current generation and are not permanently stored in the object.

You can define the values of any of the settings of all Dataproc Jobs using scripts. You do this by using :PUT_ATT commands on the Pre-Process object definition page.

For more information, see the Automic Automation product documentation:

This page includes the following:

Google Cloud Dataproc Job Parameters

The attributes available for Start Or Stop Cluster jobs, Submit jobs and Instantiate Workflow jobs are the following:

  • connection

    Defines the value of the Dataproc Connection object.

    Value: text

  • projectId

    Defines the ID of the project containing the relevant transfer configuration.

    Value: text

  • locationId

    Defines the location (region) in which your data is stored and processed.

    Value: text

  • operationType

    Defines the type of operation you want to run.

    Values for Start Or Stop Cluster jobs: Start, Stop

    • Start

      Starts the cluster.

    • Stop

      Stops the cluster.

    Values for Submit Job to cluster jobs: Submit, SubmitAsOperation

    • Submit

      Allows you to submit the job directly to the cluster (short-running jobs).

    • SubmitAsOperation

      Submits the job to the cluster with an operation id (long-running jobs).

    Values for Instantiate Workflow jobs: Instantiate, InstantiateInline

    • Instantiate

      Defines a workflow template.

    • InstantiateInline

      Defines an inline workflow that does not create or modify workflow template resources.

  • clusterName

    (Start Or Stop Cluster and Submit Jobs only)Defines when the name of the cluster that should start or stop.

    Value: text

  • jobParameters

    (Submit Jobs and Instantiate Workflow Jobs only) Defines the attributes that you want to pass to the application.

    Value: jobJsonInputType, jobJsonPath

    • jobJsonInputType

      Enter the JSON payload definition. You can also use this field if you want to include this Job in a Workflow and you want to use Automation Engine variables in it.

    • jobJsonPath

      Defines the path to the JSON file containing the attributes that you want to pass to the application.

See also:

Defining Google Cloud Dataproc Job Properties