Databricks Jobs: Run Jobs

Automic Automation Databricks Run Jobs allow you to run and monitor jobs, or run and monitor jobs using the Run now payload in your Databricks workspace from Automic Automation. Run Job requires a job that has already been created. If you invoke Create together with Run now, you can use Run submit instead, which allows you to submit your workload directly without having to create a job

On the Run Job page, you define the parameters relevant to run a job in Databricks

• Connection

  Select the Databricks Connection object containing the relevant information to connect to Databricks

  To search for a Connection object, start typing its name to limit the list of the objects that match your input.

• Execution Type:

  Define the type of execution for a job that is available for the user.

  • Run now (default)

    Runs a job and returns the Run ID of the triggered run.

  • Run submit:

    Submits a one-time run with a Databricks Run Submit JSON payload. The endpoint allows you to submit a workload directly without creating a job. Opens a new input field:

    • Run Submit JSON
      Define the parameters you want to pass on in JSON format. Ensure you define the parameters required in your Databricks environment.
      Using this method will result in the follow behavior:

      • There is no Databricks job that can be tracked in the Databricks UI.
      • The workload is submitted to the defined cluster without a job definition.

• Run Now Input Type

  Select one of the following if you use the Run now execution type:

  • Job ID with parameters

    • Run Now JSON
      If you select Job ID with parameters, you can run a job with a Databricks Run Now JSON payload. This field allows you to define all the job parameters you want to pass on in JSON format. Ensure you define the parameters required in your Databricks environment.

  • Job ID without parameters (default)

  • Job ID
    If the Job ID without parameters is selected as the Run now input type, enter your Job ID or select it from a dropdown list.

When the Job is executed, both the Job (REP) and Agent (PLOG) reports show the information that the job is running. For details, see Monitoring Databricks Jobs.

This section gives you examples of how you could define the JSON field when defining a Run Job. You have different options available.

Simple JSON Definition

The first option to define the JSON field is a simple payload definition. To do so, make sure you define the parameters required to define the job, such as the Execution Type and the Run Now Input Type.

Using Variables

You can also use variables in the payload definition.

Example

In the Run Submit JSON Parameters field, enter the following:

&CLUSTERID#

&JSON#

If the variables are not defined yet, you must define them now. You do so on the Variables page of the Run Job definition:

(Click to expand)

When you execute the Job, the variables will be replaced with the value you have just defined. This is visible in the Agent log (PLOG), see Monitoring Databricks Jobs.

Databricks Run Job in a Workflow

You can also use the JSON field if you want to include a Run Job in a Workflow and you want to use Automation Engine variables in it.

Example

In the Workflow, a Start Or Stop Cluster job starts the cluster job, and a Script object (SCRI) with the variable definition relevant for the Cluster ID and the JSON parameters precede your Run Job:

(Click to expand)

In the Run Job definition, you include the JSON variable:

(Click to expand)

When the Job is executed, the variable will be replaced with the value you have just defined. This is visible in both the Job (REP) and Agent (PLOG) reports, see Monitoring Databricks Jobs.

Example

(Click to expand)

• Defining Databricks Jobs

• Defining RA/ Integration Reports Properties

• Defining Common Job Properties

• Automic Automation / Databricks Agent Integration