Databricks Jobs: Start Or Stop Clusters

{"URL":["/*.*/awa/pa_view_pa_view_START_OR_STOP_CLUSTER_JOB_databricks"],"heroDescriptionIdentifier":"ice_hero_Databricks_Start_Stop_Cluster_Job","customCards":[{"id":"ice_specific_Databricks_Start_Stop_Cluster_Job","title":"Adding Start or Stop Cluster Job Parameters","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_Jobs_StartOrStopCluster.htm","languages":["en-us"]},{"id":"ice_RA_Integration_Report","title":"RA / Integration Reports","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_Jobs_RA_Properties.htm","languages":["en-us"]},{"id":"ice_Start_Stop_Cluster_Job_script_Databricks","title":"Setting Start or Stop Cluster Jobs through Scripts","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_Script.htm","languages":["en-us"]},{"id":"ice_related_information_Databricks_Start_Stop_Cluster_Job","title":"Related Information","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_Jobs_StartOrStopCluster.htm","languages":["en-us"]}]}

Automic Automation Databricks Start Or Stop Cluster Jobs allow you to start and stop a cluster in your Databricks environment from Automic Automation. An Databricks cluster is a set of computation resources and configurations on which you run data engineering, data science, and data analytics workloads, such as production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning.

Defining Databricks Start Or Stop Cluster Job Parameters

On the Start Or Stop Cluster Job page you define the parameters relevant to start and stop cluster jobs in Databricks.

Important!

You can decide how the JSON is created. For example, it could be created using a custom job or scripts in Automic Automation. Regardless of how the file is created, you have to make sure that it is available on the Agent machine (host).

  • Connection

    Select the Databricks Connection object containing the relevant information to connect to the application.

    To search for a Connection object, start typing its name to limit the list of the objects that match your input.

  • Operation Type

    Select the Operation Type to start or stop the cluster on Databricks. You can select Start or Stop:

    • Start (default)
      Starts the specified cluster.

      Note: To be able to run a job, the cluster assigned to it must be in start mode or running mode. If it is not, a message is printed to the log.

    • Stop
      Stops the specified cluster.

  • Cluster ID

    Enter your Cluster ID or select the Cluster ID and Name from a dropdown list.

When the Job is executed, both the Job (REP) and Agent (PLOG) reports show the information that the cluster is running or has been destroyed. For details, see Monitoring Databricks Jobs.

Note:

The Pre-Process page allows you to define the settings of the Jobs using script statements. These statements are processed before the Run Job or Start or Stop Cluster Job is executed, see Databricks Jobs: Setting Job Properties Through Scripts.

See also: