Monitoring AWS Glue Jobs

When you execute an object in Automic Automation, the corresponding task is visible in the list of Tasks in the Process Monitoring perspective. The Process Monitoring gives you the full range of possibilities to monitor, analyze, identify problems and remediate them. It displays comprehensive data on active and inactive tasks, it provides tools to filter and group tasks. In this perspective, you can modify active tasks, open their reports and execution lists and troubleshoot.

If you want to learn more about how to monitor your jobs, please refer to the Automic Automation product documentation at Monitoring Tasks.

For more information on how to work with tasks on the Process Monitoring perspective, refer to the Automic Automation documentation at Working with Tasks.

Monitoring the Job Details

1. Go to the list of tasks in the Process Monitoring perspective.

2. Find the AWS Glue task.

3. Select the task and click the Details button.

   The Details pane opens on the right hand side and shows a summary of the execution of the selected task. The General and Job sections of the Details pane display information about the object configuration and its execution.

   The Object Variables section displays information that the AWS Glue system reports back to Automic Automation for the following jobs:

   • Job Run jobs: &JOB_RUN_ID#

   • Blueprint jobs: &BLUEPRINT_RUN_ID#

   • Workflow jobs: &JOB_WORKFLOW_ID#

Canceling and Restarting Jobs

Start Trigger and Start Blueprint jobs cannot be canceled nor restarted.

When working with Start Crawler, Start Job Run and Start Workflow jobs, you can execute them and cancel their corresponding tasks in the Process Monitoring perspective.

Canceling means that the Automic Automation task is canceled as well as the job on the AWS Glue environment. If for some reason canceling the execution fails - for example, if you do not have the rights to cancel the execution - the execution remains active and that information is visible on the job report.

Additionally, when working with Start Job Run jobs, you can also restart a task that was canceled or that failed. Restarting a task in Automic Automation allows you to restart the task also in your AWS Glue application.

For more information on how to work with tasks on the Process Monitoring perspective, refer to the Automic Automation documentation at Working with Tasks.

Agent Connection

If the Agent stops working or loses connection, the job execution stops, but resumes as soon as the Agent is connected again.

Reports

When jobs are executed, Automic Automation generates output files and reports that you can open from the user interface. Combined with the Execution Data, these reports provide comprehensive information about the status and the history of your executions. This information tracks all processes and allows you to control and monitor tasks, ensuring full auditing capability.

Reports are kept in the system until they are explicitly removed by a reorganization run. During a reorganization run, the Execution Data can be kept. Keeping the execution data is an advantage, because reports can consume a large amount of hard drive space. Removing the reports from your database does not mean loss of important historical data.

The following reports provide a comprehensive insight into the executions of your jobs:

• Agent log (PLOG)

  This report captures everything that has to do with the Job execution from the Automic Automation perspective: When the Job started, the Job configuration parameters, whether there has been any validation and its results, and so on. It also contains the response received from the target system but in a format that cannot be parsed for further processing.

  Since this report logs all that happens during the execution, it is your first place to go if you need to troubleshoot issues, also if those issues have to do with the configuration of the Job. For example, if you use variables to populate the configuration fields, these variables are resolved at runtime. This report show the actual variable values and not their names.

• Report (REP)

  This report captures the information that the target system sends to Automic Automation when the Job is executing/has executed. It contains all relevant information about the Job execution and its results. It provides this information in a format that can be easily parsed for further processing (JSON, XML and so on).

  Note:

  The Report (REP) cannot be parsed if the job throws and exception. In this case, only the exception message is written into both the Report (REP) and Agent (PLOG) logs.

  You will most probably include the Jobs in other Automic Automation objects such as Workflows or Schedules to orchestrate operations. When you do so, you can use the content of this report on the Post Process pages of the Job definition to trigger further actions based on that content. For more information see, Process Pages in the Automic Automation product documentation.

• Directory

  This report provides the list of job output files that the job has created for this particular execution and that are available for downloading.

To Access the Reports

Do one of the following:

• From the Process Assembly perspective

  • After executing a Job, a notification with a link to the report is displayed. Click it to open the Last Report page.

  • Right-click a job and select Monitoring > Last Report.

  In either case, the most recent report that is created for the object opens up.

• From the Process Monitoring perspective

  • Right-click a task and select Open Report.

    The report of that particular execution opens up. Its runID is displayed in the title bar to help you identify it.

  • In the list of Executions for a particular task, right-click and select Open Report.

See also:

• Automic Automation / AWS Glue Agent Integration

• Defining AWS Glue Job Properties

• Executing AWS Glue Jobs