Setting Up the Airflow Connector

The AAI integration for Airflow supports Apache Airflow and Google Cloud Composer. It allows you to put the runtime information of your DAG job executions from different Airflow instances into a single view. This means that you can see running executions (and possible failed ones) of more than one Airflow instance in a single interface, without having to switch from one to another.

This topic gives you an overview of the AAI integration for Airflow and instructions on how to configure your Airflow Connector in Windows and UNIX.

Overview

Click the image to expand it.

Regardless of the Airflow provider that you want to integrate with AAI, you only require one Airflow Connector. Each instance of the respective Airflow provider requires its own Scheduler. Several schedulers can be linked to one Connector. However, only one Connector can be linked to an AAI environment. For more information, see Adding/Editing/Deleting Airflow Schedulers.

Before adding an Airflow instance to your AAI system, you must have AAI up and running. For more information, see Installation and Upgrade.

The naming conventions for the Airflow Connector .zip file are AAI.Airflow_Connector_<product_version>.zip.

If you need help with the installation, please contact our support team at https://support.broadcom.com/.

Also, you can secure your Connectors using TLS 1.2 based secure connections. For more information, see Securing AAI

Installing/Upgrading the Airflow Connector on Windows

To install or upgrade the Airflow Connector on Windows:

1. Go to https://downloads.automic.com/, log in with your Broadcom account and download the relevant version of the Connector. For more information ,see Downloads.

2. Run the AAI-airflow-connector.exe file to setup the installer.

3. Follow the wizard's instructions. You can define a destination directory, if you want to create a Start Menu folder and / or shortcuts for all users.

   Make sure you enter the information required:

   • AAI server URI

     You must specify the protocol, domain host name or IP address, and a listening port number of the AAI server. This information should be captured during the AAI server installation.

     If you are using the Connector certificate, this URI may use HTTPS and port 4443. Otherwise, it will likely be the same port that is used to launch the AAI client from a browser, for example http{s}://hostname:port.

   • Connector name

     Name of your choice. This name will be selected from the web UI when adding a scheduler to AAI.

   • Host name

     Domain name or IP address of the host where the Connector runs

   • Connector key password

     This is only required if an HTTPS connection is used between the Connector and the AAI server. If HTTP is used, leave this empty and hit Enter.

   • AAI server user name

     Only required when using HTTP.

     Name of the user ID that will be used to establish a connection to the AAI server. The user must be authorized to add new schedulers.

   • AAI server password

     This is required only if a HTTP connection is used between the Connector and the AAI server. If HTTPS is used, this can be omitted.

   • Connector domain

     Typically this is JAWS. This is the login domain for the user/password provided and it can be any of the list of domains that have been set up in AAI. For more information about domains, see Domains.

   • Log file output path

     The path to the location of the log files. The directory must exist and be writable for the connector user account.

   • Max log file size

     The maximum size of the log files. Must be defined as an integer in MB and must be 1 or larger.

   • Max log file count

     The number of log files to be retained. Must be defined as an integer in MB and must be 5 or larger.

   When you are done with the wizard, you can find the Airflow Connector folder in the directory that you defined.

   The Connector starts as a Service (AAI-airflow-connector).

   If the connection is successful, the application.yml file is created in the folder where you have installed the Connector. This is a configuration file that contains the configuration values that you have just entered. At the same time, a backup copy of the application.yml is automatically created and renamed to application.yml.bak.

   In the case of a failure, the application.yml file is not populated with the values you entered. You would have to manually repeat the configuration process.

You also have to add Airflow schedulers to your AAI environment. To do so, go to System > Schedulers in your web interface and follow the steps in the dialog to add a Airflow scheduler. For more information, see Adding/Editing/Deleting Airflow Schedulers.

If the process was successful, the new schedulers are listed with all the schedulers available in your system.

Starting, Stopping, and Connector Status

You can also run the installation file to start or stop the connector or to check its status.

.\aai-airflow-connector.exe /start

.\aai-airflow-connector.exe /stop

.\aai-airflow-connector.exe /status

Expected output if the service is running:

Running.

Auto-start.

Expected status if the service is not running:

Not running.

Auto-start.

For details, refer to the output.log file that is created in the folder where the Connector was installed.

Installing the Airflow Connector on UNIX

To install the Airflow Connector on UNIX:

1. Go to https://downloads.automic.com/, log in with your Broadcom account and download the relevant version of the AirflowConnector. For more information ,see Downloads.

2. Unzip the .zip file that you downloaded and unpack the .tar.gz file within to get the connector-install.sh and connector.AAI-airflow-<version>.rpm files.

3. Run connector-install.sh to start the installation, for example:

   ./connector-install.sh install

   The Connector is installed using the .rpm file provided.

4. Configure the Connector to point to your AAI installation. To do so, run the configure script:

   cd /opt/AAI-airflow-connector

   sudo -u AAI-airflow-connector./configure-AAI-airflow-connector.sh

   Follow the prompts and make sure you enter the information required:

   • AAI server URI

     You must specify the protocol, domain host name or IP address, and a listening port number of the AAI server. This information should be captured during the AAI server installation.

     If you are using the Connector certificate, this URI may use HTTPS and port 4443. Otherwise, it will likely be the same port that is used to launch the AAI client from a browser, for example http{s}://hostname:port.

   • Connector name

     Name of your choice. This name will be selected from the web UI when adding a scheduler to AAI.

   • Host name

     Domain name or IP address of the host where the Connector runs

   • Connector key password

     This is only required if an HTTPS connection is used between the Connector and the AAI server. If HTTP is used, leave this empty and hit Enter.

   • AAI server user name

     Only required when using HTTP.

     Name of the user ID that will be used to establish a connection to the AAI server. The user must be authorized to add new schedulers.

   • AAI server password

     This is required only if a HTTP connection is used between the Connector and the AAI server. If HTTPS is used, this can be omitted.

   • Connector domain

     Typically this is JAWS. This is the login domain for the user/password provided and it can be any of the list of domains that have been set up in AAI. For more information about domains, see Domains.

   • Log file output path

     The path to the location of the log files. The directory must exist and be writable for the connector user account.

   • Max log file size

     The maximum size of the log files. Must be defined as an integer in MB and must be 1 or larger.

   • Max log file count

     The number of log files to be retained. Must be defined as an integer in MB and must be 5 or larger.

   Examples

   These examples show the Connector installed running HTTPS and on the default port. The Connector name can be freely defined and shows up in the UI when adding the scheduler.

   The first example does not include Java on the Linux $PATH variable:

   # sudo -u AAI-airflow-connector ./configure-AAI-airflow-connector.sh
   Java not found on the system PATH.
   If Java is not installed, please exit and install it.
   Otherwise, please enter the full path of the java executable: 

   At this point, the configuration script waits for you to enter a valid path to a Java executable and keeps re-prompting until a valid path is entered.

   Otherwise, please enter the full path of the java executable: /bin/java
   
   **********
   After configuring the connector, please run the following command to configure
   Java for the system service as well:
   $ sudo ./configure-service.sh
   **********

   The message above is displayed when you take this path. You will need to run the additional configuration script and enter the same Java path as above so that the configured service files are updated appropriately.

   From this point onwards, the Connector setup is the same for installations with and without Java defined on the Linux $PATH variable.

   If you are planning to make a TLS based connection to AAI then before proceeding
   please obtain the connector certificate package from the AAI server
   and copy it to ./secure/
   Press ENTER to continue...
   
   *** CONFIGURING CONNECTOR ***
   
   Backing up configuration file to: application.yml.bak...
   ...Configuration file backed up to: application.yml.bak.
   
   Please enter the following configuration parameters.
   
   AAI server URI [https://localhost:4443]: https://<host>:4443
   Connector name: Airflow-GCC
   Host name: host
   Connector key password (will not echo, required for HTTPS):
   AAI server username (required for HTTP):
   AAI server password (will not echo, required for HTTP):
   Connector domain: JAWS
   Log file output path. The directory must exist and be writable for the connector user account. [./logs]:
   Max log file size. An integer, in MB. Must be >= 1. [10]:
   Max log file count. An integer, in MB. Must be >= 5. [5]:

   The second example includes Java on the Linux $PATH variable:

   [sudo -u AAI-airflow-connector ./configure-AAI-airflow-connector.sh
   If you are planning to make a TLS based connection to AAI then before proceeding
   please obtain the connector certificate package from the AAI server
   and copy it to ./secure/
   Press ENTER to continue...
   
   *** CONFIGURING CONNECTOR ***
   
   Backing up configuration file to: application.yml.bak...
   ...Configuration file backed up to: application.yml.bak.
   
   Please enter the following configuration parameters.
   
   AAI server URI [https://localhost:4443]: https://<host>:4443
   Connector name: Airflow-GCC
   Host name: host
   Connector key password (will not echo, required for HTTPS):
   AAI server username (required for HTTP):
   AAI server password (will not echo, required for HTTP):
   Connector domain: JAWS
   Log file output path. The directory must exist and be writable for the connector user account. [./logs]:
   Max log file size. An integer, in MB. Must be >= 1. [10]:
   Max log file count. An integer, in MB. Must be >= 5. [5]:

   In both cases, if the installation ended successfully you see the following:

   Verifying AAI server connectivity (please stand by)...2020-12-08 23:54:55,708 INFO [ApiCallerUtil] Attempting HTTPS connection to AAI
   
   ...AAI server connectivity verified.
   
   Saving connector configuration to: application.yml...
   ...connector configuration saved to: application.yml.
   
   *** CONNECTOR CONFIGURED ***

5. Start the service using the following command:

   sudo systemctl start AAI-airflow-connector

You also have to add Airflow schedulers to your AAI environment. To do so, go to System > Schedulers in your web interface and follow the steps in the dialog to add a Airflow scheduler. For more information, see Adding/Editing/Deleting Airflow Schedulers.

If the process was successful, the new schedulers are listed with all the schedulers available in your system.

Starting, Stopping, Upgrading, Uninstalling and Connector Status

You can verify the service by running the following command:

sudo systemctl status AAI-airflow-connector

You can also use the following systemclt commands to stop or to restart the connector, or to get its status. For example:

sudo systemctl stop AAI-airflow-connector

sudo systemctl restart AAI-airflow-connector

sudo systemctl status AAI-airflow-connector

If for any reason the service is disabled, you can use the following systemctl command to enable it:

sudo systemctl enable AAI-airflow-connector

This is recommended to have the Connector restart automatically as part of the server reboot process.

You can also run the connector-install.sh to upgrade or uninstall the connector, or to get its status using the following commands:

./connector-install.sh upgrade

./connector-install.sh uninstall

./connector-install.sh status

If the current status of your system does not allow you to run a specific command, you get a corresponding error message. For example, you cannot upgrade a package that has not yet been installed.

./connector-install.sh upgrade

Can't upgrade the connector 'connector.airflow" isn't installed. Please 'install' instead.

See also:

• Connectors for Universal Schedulers

• Configuring the Connectors for TLS Authentication

• Adding/Editing/Deleting Airflow Schedulers