Automic Automation / Airflow Agent Integration

BROADCOM's Airflow Agent Integration lets you orchestrate, execute and monitor the execution and status transitions of Apache Airflow jobs (DAG Jobs) that you run on your cloud platforms. The Airflow Agent establishes the connection between Automic Automation and the cloud solution on which the DAGs run. The Agent starts the execution of these jobs and makes both their monitoring and the corresponding reporting possible.

The Airflow Agent Integration gives you full access to Automic Automation's powerful workload automation, scheduling and monitoring capabilities. It orchestrates Workflows from various cloud solutions, such as Apache Airflow and the Google Cloud Composer (GC Composer), and Automic Automation objects. You can build Automic Automation Workflows that combine Apache Airflow, GC Composer and Automic Automation objects to orchestrate processes in hybrid environments.

Once the Airflow Agent is installed, both the Agent and all necessary components are configured (Connection object, Run DAG Jobs) and your Automic Automation Workflows are defined, you can start executing, working with and monitoring your cloud processes directly from Automic Automation. Automic Automation pulls the information from your cloud solution and knows which tasks are included in the DAG job. In Automic Automation's Process Monitoring perspective you can access the Workflow monitor, where all the tasks included in the DAG jobs and their remote statuses are displayed. When the DAG ends successfully in the cloud environment, the Workflow monitor shows this information too. Automic Automation also provides downloadable execution reports.

Airflow Integrations

The Airflow Agent provides integrations with the following cloud solutions:

  • Apache Airflow

  • Google Cloud Composer (GC Composer)

Architecture of the Airflow Agent Integration

Screenshot that illustrates the Architecture f the Agent in relation with Automic Automation and the clour solutions

The Airflow Agent communicates with Automic Automation through the JCP, which is one type of Automic Automation server processes. The Airflow Agent communicates with the cloud solution through Connection objects that contain all the necessary connection parameters (including credentials). This communication is based on the REST APIs provided by the cloud solution. The Agent can be installed on any machine in your environment or as a pod in Kubernetes or docker container.

For more information about the Automic Automation server processes, see Multi-Server Operations in the Automic Automation product documentation.

Release Cycles

The release cycles of the Airflow Agent Integration are independent of the Automic Automation ones.

The Airflow Agent Integration orchestrates processes across multiple cloud platforms. The release cycles for the integrations with each of those cloud platforms are also independent of one another.

For more information, see Compatibility Information.

Important Consideration

To be able to implement (install and configure) and work with the Agent Integration, a sound knowledge of Automic Automation is required. If you are new to Automic Automation, we strongly recommend that you take the following courses before beginning with this integration:

For all users:

For administrators:

For developers and object designers

This documentation describes how to set up the integration and how to work with Automic Automation Run DAG Jobs: