Establishing the Connection to the Airflow Environment

{"URL":["/*.*/awa/pa_view_pa_view_CONN_airflow"],"heroDescriptionIdentifier":"ice_hero_CONN_Airflow","customCards":[{"id":"ice_Airflowspecific_CONN","title":"Defining the Connection Parameters for Apache Airflow","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AIRFLOW/*.*/Agent%20Guide/Content/Airflow/Airflow_Connection.htm","languages":["en-us"]},{"id":"ice_GCCspecific_CONN","title":"Defining the Connection Parameters for Google Cloud Composer (GCC)","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AIRFLOW/*.*/Agent%20Guide/Content/Airflow/Airflow_Connection.htm","languages":["en-us"]},{"id":"ice_GoogleCloudStorage_S3_Authentication_video_CONN","title":"Watch the Video: Authentication Methods","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AIRFLOW/latest/Agent%20Guide/Content/_CommonTopics/Video_GoogleAuthMethods.htm","languages":["en-us"]},{"id":"ice_video_Airflow","title":"Watch the Airflow Video","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AIRFLOW/latest/Agent%20Guide/Content/Airflow/Airflow_Video.htm","languages":["en-us"]},{"id":"ice_related_information_CONN_Airflow","title":"Related Information","type":"customize","url":"https://docs.automic.com/documentation/webhelp/english/ALL/components/IG_AIRFLOW/*.*/Agent%20Guide/Content/Airflow/Airflow_Connection.htm","languages":["en-us"]}]}

The communication between the Airflow Agent and the target system is established by Connection objects. They contain the parameters (target system endpoint, login data and so on) required to authenticate on and connect to the target cloud solution. As an administrator user, you create the necessary Airflow Connection objects. As a developer or object designer, you assign the respective Airflow Connection object to the Automic Automation Run DAG Jobs to execute and monitor the jobs on the target cloud solution without leaving Automic Automation. This topic explains how to configure the Airflow Connection object.

The Airflow Agent Integration supports the following authentication methods:

  • For Apache Airflow

    • Basic

    • OAuth2

  • For the Google Cloud Composer

    • Service account key

    • Workload identity

Tip:

To configure an Automic Automation Airflow Connection object you will need the authentication data (credentials, endpoints, tokens and so forth) that enable the login to the target cloud solution. You can get this data from the team in your organization that is responsible for maintaining the target cloud solution.

The Connection Object definition consists of an Agent-specific page and pages that are common to all Connection objects.

Adding an Airflow Connection Object

  1. In the Automic Web Interface, go to the Process Assembly perspective. It opens to the Explorer view that contains the list of jobs that are available to you in your system.

  2. You have two options:

    • Right-click anywhere on the list and select Add > Add Object.

    • Click the Add Object button on the toolbar.

  3. On the Add Object dialog, start typing Airflow in the Search field.

  4. Select AIRFLOW Connection and click Add. The Object Name dialog is displayed.

  5. Enter a descriptive Name.

  6. Optionally, enter a short and descriptive Title that helps you recognize the purpose of the object.

  7. Click OK. A new page opens where you can start with the object definitions.

AIRFLOW Page

On this page you specify the parameters that are specific to the cloud solution that you want to connect to.

Connecting to an Apache Airflow Target

The Agent integration with Apache Airflow supports two authentication methods, Basic and OAuth2.

To Connect to an Apache Airflow Environment

  1. Specify the Endpoint that identifies the environment that you want to connect to.

    Example: http://<hostname>:<port>/api/v1

  2. In Provider select Apache Airflow.

  3. Select the Authentication Type used to authenticate to the target environment. Reach out to the team that maintains the credentials for information about them.

    • Basic

      Credentials required to log in to Apache Airflow.

      1. Username

        The name of the user with privileges to access the Apache Airflow environment.

      2. Password

        Password of the Apache Airflow user.

    • OAuth2

      1. OAuth2 Endpoint

        URL that clients call to request the token.

      2. Client ID

        Identifier as defined in your OAuth2 system.

      3. Client Secret

        Encrypted secret value as defined in your OAuth2 system.

      4. Scope

        Range of applications that the token will grant access to.

  4. Save the object.

Connecting to a Google Cloud Composer Target

The Agent integration with Apache Airflow supports two authentication types:

  • Service account key

  • Workload identity if you have an AAKE environment and both AAKE and the Agent run in a Kubernetes cluster

To Connect to an Google Cloud Composer Environment

  1. In Endpoint specify the URL that identifies the network address of the Google Cloud Composer environment.

  2. In Provider select Google Cloud Composer.

  3. In Authentication Type select one of the following:

    • Service Account Key

      In Service Account Key enter the file path to the JSON file that contains the authentication information.

      When creating a service account in the Google Cloud Composer, a public key can be created which provides the JSON file. Contact the team in your organization that is responsible for maintaining the Google Cloud Composer environment as they will be able to provide you with the data that you need.

    • Workload Identity

      Select this option if you have an AAKE environment and the Airflow Agent and AAKE run in a Kubernetes cluster.

      With this option you do not have to enter anything else.

      For more information about containerized Agents in AAKE, see Installing Containerized Java Agents in the Automic Automation documentation.

  4. Save the object.

Common Definition Pages

In addition to the Agent-specific connection parameters, you can also specify optional properties. You do so on the following definition pages:

Next Step

As soon as the Connection object is configured, developers and object designers can select it when defining the Run DAG Jobs.

See also: