Monitoring Apache Kafka Jobs

When you execute an object in Automic Automation, the corresponding task is visible in the list of Tasks in the Process Monitoring perspective. Process Monitoring gives you the full range of possibilities to monitor, analyze, identify, and remediate problems. It displays comprehensive data on active and inactive tasks, it provides tools to filter and group tasks. In this perspective, you can modify active tasks, open their reports, and execution lists and troubleshoot.

If you want to learn more about how to monitor your jobs, please refer to the Automic Automation product documentation at Monitoring Tasks.

For more information on how to work with tasks on the Process Monitoring perspective, refer to the Automic Automation documentation at Working with Tasks.

This page includes the following:

Monitoring Apache Kafka Job Details

Go to the list of tasks in the Process Monitoring perspective.
Find the Apache Kafka task.
Select the task and select the Details button.

The Details pane opens on the right hand side and shows a summary of the execution of the selected task. The General and Job sections of the Details pane display information about the object configuration and its execution in Automic Automation.

The Object Variables section displays the information that the Apache Kafka system reports back to Automic Automation:
- &KAFKA_MESSAGE_HEADERS#
- &KAFKA_MESSAGE_KEY#
- &KAFKA_MESSAGE_VALUE#

Canceling Apache Kafka Jobs

You can execute Jobs and cancel the corresponding tasks in the Process Monitoring perspective.

Canceling means that both the Automic Automation task and the job on the Apache Kafka environment are canceled. If canceling the execution fails (for example, if you do not have the rights to cancel the execution), the execution remains active and that information is visible on the job report.

For more information on how to work with tasks on the Process Monitoring perspective, refer to the Automic Automation documentation at Working with Tasks.

Apache Kafka Agent Connection

If the Agent stops working or loses connection, the Job execution stops but resumes as soon as the Agent is connected again.

Apache Kafka Reports

When jobs are executed, Automic Automation generates output files and reports that you can open from the user interface. Combined with the Execution Data, these reports provide comprehensive information about the status and the history of your executions. This information tracks all processes and allows you to control and monitor tasks, ensuring full auditing capability.

Reports are kept in the system until they are explicitly removed by a reorganization run. During a reorganization run, the Execution Data can be kept. Keeping the execution data is an advantage, because reports can consume a large amount of hard drive space. Removing the reports from your database does not mean loss of important historical data.

The following reports provide a comprehensive insight into the executions of your jobs:

Agent log (PLOG)

This report captures everything that has to do with the Job execution from the Automic Automation perspective: When the Job started, the Job configuration parameters, whether there has been any validation and its results, and so on, It also contains the response received from the target system but in a format that cannot be parsed for further processing.

Since this report logs all that happens during the execution, it is your first place to go if you need to troubleshoot issues, also if those issues have to do with the configuration of the Job. For example, if you use variables to populate the configuration fields, these variables are resolved at runtime. This report show the actual variable values and not their names.
Report (REP)

This report captures the information that the target system sends to Automic Automation when the Job is executing/has executed. It contains all relevant information about the Job execution and its results. It provides this information in a format that can be easily parsed for further processing.

When Consumer Jobs are completed, the original message in JSON format is written to this report. The JSON structure in the report is as follows:
```
{
 "message_value": "value",
 "message_key": "key",
 "message_headers": {
 "key1": "value1"
 }
}
```
You will most probably include the Jobs in other Automic Automation objects such as Workflows or Schedules to orchestrate operations. When you do so, you can use the content of this report on the Post Process pages of the Job definition to trigger further actions based on that content. For more information see, Process Pages in the Automic Automation product documentation.
Directory

This report provides the list of job output files that the job has created for this particular execution and that are available for downloading.

To Access the Reports

Do one of the following:

From the Process Assembly perspective
- After executing a Job, a notification with a link to the report is displayed. Click it to open the Last Report page.
- Right-click a job and select Monitoring > Last Report.
In either case, the most recent report that is created for the object opens up.
From the Process Monitoring perspective
- Right-click a task and select Open Report.
  
  The report of that particular execution opens up. Its runID is displayed in the title bar to help you identify it.
- In the list of Executions for a particular task, right-click and select Open Report.

To Access the Reports

Do one of the following:

From the Process Assembly perspective
- After executing a Job, a notification with a link to the report is displayed. Click it to open the Last Report page.
- Right-click a job and select Monitoring > Last Report.
In either case, the most recent report that is created for the object opens up.
From the Process Monitoring perspective
- Right-click a task and select Open Report.
  
  The report of that particular execution opens up. Its runID is displayed in the title bar to help you identify it.
- In the list of Executions for a particular task, right-click and select Open Report.