Establishing the Connection to Databricks

{"URL":["/*.*/awa/pa_view_pa_view_CONN_databricks"],"heroDescriptionIdentifier":"ice_hero_CONN_Databricks","customCards":[{"id":"ice_Azure_specific_CONN_Databricks","title":"Defining the Connection Parameters for Azure","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_EstablishingConnection.htm","languages":["en-us"]},{"id":"ice_AWS_specific_CONN_Databricks","title":"Defining the Connection Parameters for AWS","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_EstablishingConnection.htm","languages":["en-us"]},{"id":"ice_Proxy_CONN_Databricks","title":"Defining the Proxy Parameters","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_EstablishingConnection.htm","languages":["en-us"]},{"id":"ice_related_information_CONN_Databricks","title":"Related Information","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_DATABRICKS/*.*/Agent%20Guide/Content/Databricks/Databricks_EstablishingConnection.htm","languages":["en-us"]}]}

A Connection object contains the parameters that make the communication between the Agent and the target system possible. These parameters (target system endpoint, login data and so on) are required to authenticate on and connect to the target cloud solution.

As a developer or object designer, you assign the respective Databricks Connection object to the respective Run Jobs or Start or Stop Cluster Jobs to create and execute them on the Databricks system without leaving the Automic Web Interface.

As an administrator user, you create the Databricks Connection objects that provide the log in data required to establish the connection to the Azure Databricks or AWS Databricks system.

Tip:

To configure an Automic Automation Databricks Connection object you will need the authentication data (credentials, endpoints, tokens and so forth) that enable the login to the target cloud solution. You can get this data from the team in your organization that is responsible for maintaining the target cloud solution.

The Connection Object definition consists of an Agent-specific page and other pages that are common to all Connection objects.

This page includes the following:

Adding a Databricks Connection Object

In the Automic Web Interface, go to the Process Assembly perspective. It opens to the Explorer view which contains the list of jobs that are available to you in your system.
You have two options:
- Right-click anywhere on the list and select Add > Add Object.
- Click the Add Object button on the toolbar.
On the Add Object dialog, start typing the name of the relevant integration in the Search field.
Select the corresponding Connection object and click Add. The Object Name dialog is displayed.
Enter a descriptive Name.
Optionally, enter a short and descriptive Title that helps you recognize the purpose of the object.
Click OK. A new page opens where you can start with the object definitions.

Databricks Page

This page contains a Connection and a Proxy section.

The definition of the Connection object depends on the Cloud Type that you select. You can prepare the Connection object for Databricks with the following options:

Azure (default) to connect to an Azure endpoint
AWS to connect to an AWS endpoint

Connecting the Automation Engine to the Azure Databricks Agent

In this section, you define the parameters relevant for the connection to the Azure endpoint.

Make sure you select Azure as your Cloud Type and continue with the definition of the object.

(Click to expand)

Workspace URL

The workspace URL endpoint of Databricks that identifies the database instance you must create to run a job.
Authentication Type

The authentication types available are Service Principal, Personal Access Token, and Token from File.

Depending on the authentication type that you select, define the following:
- Service Principal (default)
  - Azure AD URL
    
    URL that identifies the network address of the Azure AD used to secure the application.
    
    By default, the URL (https://login.microsoftonline.com) points to the Azure AD log in. If you do not want to use this URL, make sure you change the default definition.
  - Tenant ID
    
    Identifier of the OAuth2 Azure AD tenant.
  - Client ID
    
    Identifier of the application in the OAuth2 Azure portal.
  - Client Secret
    
    OAuth2 client secret. Encrypted secret value used as the application password.
  - OAuth2
    
    Version of the OAuth2 authentication method. The versions available are v1.0 (default) and v2.0.
    
    Depending on the version selected, you need to define either the Resource (v1.0) or the Scope (v2.0).
  - Resource / Scope
    
    URL of the application within the Databricks to grant access to.
    
    The default URL points to the Azure Databricks log in.
    - Resource: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
      The OAuth2 resource. This field only appears if you select OAuth2 as the authentication method and the OAuth2 method is v1.0.
    - Scope: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default
      The OAuth2 scope. This field only appears if you select OAuth2 as the authentication method and the OAuth2 method is v2.0.
    If you do not want to use the default URL, make sure you change the definition.
- Personal Access Token
  - Token: The personal access token used to log into the Azure Databricks environment. To generate the personal access token, see Authentication for Azure Databricks automation - Azure Databricks | Microsoft Learn.
- Token From File
  - Token File Path: Provide the path to the file in which the bearer token is stored on the agent computer.

Connecting the Automation Engine to the AWS Databricks Agent

In this section, you define the parameters relevant for the connection to the AWS endpoint.

Make sure you select AWS as your Cloud Type and continue with the definition of the object.

(Click to expand)

Workspace URL

The workspace URL endpoint of Databricks that identifies the database instance you must create to run a job.
Authentication Type

The available authentication type is Personal Access Token
Token

The personal access token used to log into the AWS Databricks environment.

Proxy Section

If the Databricks system is behind a proxy server, you define the parameters relevant for the connection to that system in this section.

Proxy Host Name

Host name or IP address of the proxy server to which you want to connect.
Proxy Port

Port used by the proxy server.
Proxy Username

User name used to authenticate the proxy server.
Proxy Password

Password of the user used to authenticate the proxy server.

Common Definition Pages

In addition to the Agent-specific connection parameters, you can also specify optional properties. You do so on the following definition pages: