What's New in 21.1.0

This section provides information about the new features and enhancements that have been implemented in the Databricks Agent integration.

Databricks Template Object Names have changed

The following objects were renamed in order to reflect the fact that this integration now works with Azure Databricks and AWS Databricks:

• The Azure Databricks Agent was renamed to Databricks Agent.

• Azure Databricks Jobs are now available as Databricks Jobs.

• Azure Databricks Connection objects are now available as Databricks Connection objects.

We recommend you create new objects for this version and not reuse existing Azure Databricks objects of the first release. For details, see Checking for Incompatibilities

New Version of the Databricks Agent

The Databricks Agent has been renewed. We recommend that you delete your existing agent installation and start a new installation of the Agent. For details, see Checking for Incompatibilities

The new installation process is easy, you download the installation package, rename and adapt the INI file to fit your environment and start the agent. For more information, see Installing the Agent for Databricks

Support of AWS Endpoint for Databricks Integration

As of this version, the Databricks Agent allows you to integrate Databricks jobs on an AWS endpoint. This enhanced Agent allows to schedule Databricks pipelines and monitor their status, using the public Databricks REST API. The functionality of the Databricks Connection object has been enhanced for this purpose. For more information, see Establishing the Connection to Databricks

Once connected to your AWS endpoint you can run all the Databricks jobs also in Databricks.