Defining the Run DAG Job Properties

An Automic Automation Run DAG job represents a Run DAG job on the cloud environment with which it communicates through the Connection object. When you execute the Automic Automation Run DAG Job, you trigger a run of the DAG on the target cloud environment. You can then monitor its execution from Automic Automation too. You define the Automic Automation Run DAG job on various pages. This topic explains how to configure the Run DAG Job page.

• Connection

  Connection object that establishes the communication between the Agent Integration and the target cloud solution.

• DAG ID

  DAG in the target cloud solution that this Automic Automation Run DAG Job will execute. You have two options:

  • Enter the ID of the DAG in the input field.

  • Click the browser button to open a dialog where you can search for and select the DAG. The list in this dialog displays all the DAGs available in the target cloud solution. By default, this list shows 50 DAGs; you can extend the list to up to 200 DAGs by entering a number in Maximum Results.

    Select one DAG from the list or search for it by typing its name or part of its name in DAG ID. If you type part of the name and click Filter, the list only the DAGs that match the entered string.

• Configuration JSON

  JSON file that contains the configuration of the DAG on the target cloud solution. Ask the team that maintains the cloud environment to provide it to you.

  You only need to enter it here if you want to modify some of its parameters. This is the case if, for example, if you want to include this Job in a Workflow and you want to use Automation Engine variables in it.

  Example:

  A DAG on your Apache Airflow solution has the following configuration:

  from datetime import datetime
  from airflow import DAG
  from airflow.operators.bash import BashOperator
  
  dag = DAG("echo", schedule_interval=None, start_date=datetime(2017, 3, 20), catchup=False)
  
  echo_task = BashOperator(
      task_id='echo_task',
      bash_command="echo value: " "{{ dag_run.conf['conf1'] }}",
      dag=dag,
  )

  You want to replace conf1 with an Automation Engine variable called &ECHO#.

  In the Configuration JSON field you enter the following:

  {"conf1":"&ECHO#"}

  If the variable is not defined yet, you must define it now. You do it on the Variables page of the DAG Job definition:

  (Click to expand)

  

  As a result, when you execute the Run DAG Job in Automic Automation, it will also execute on the Apache Airflow solution and the variable will be replaced with the value you have just defined. This is visible in the Apache Airflow log for the job:

  (Click to expand)

  

   

• Fail on Paused DAG

  By default, this option is selected. This means that the status of the Automic Automation Job will be ENDED_NOT_OK when the status of the DAG on the cloud solution is paused.

  If you deselect this option, when the Airflow DAG is paused, the status of the Automic Automation Run DAG Job is set to queued.

• Show Child Tasks In Process Monitoring

  If selected, when you execute this Job both the parent (the DAG) and its children (the tasks in the DAG) are available and visible in the Tasks list in the Process Monitoring perspective.

  If you do NOT select this option, the Tasks list displays the DAG task only. When you execute the Job and open its report, it shows a summary of the child tasks that are available in the Airflow DAG. It also contains the logs of the individual tasks.

• Logging Level

  • Never (default)

    If selected, no logs will be available after executing the Job.

  • Child tasks

    If selected, the logs of the child tasks of the DAG job will be available after executing the Job. Be aware that selecting this option might lead to very long reports and it might impact the performance of your system.

• Retrieve Child Task Logs

  This option is only visible if you have previously selected Select Logging Level: DAG and child tasks. You define here when the child task logs will be written to the Job Report:

  • On error only (default)

    The logs are written to the Job Report only when the DAG and/or any of its tasks fail.

  • Always

    The logs are written always, regardless of the execution status of the DAG and/or its tasks.

• Defining Run DAG Jobs