Monitoring S3 Jobs

When you execute an object in Automic Automation, the corresponding task is visible in the list of Tasks in the Process Monitoring perspective. Process Monitoring gives you the full range of possibilities to monitor, analyze, identify, and remediate problems. It displays comprehensive data on active and inactive tasks, it provides tools to filter and group tasks. In this perspective, you can modify active tasks, open their reports, and execution lists and troubleshoot.

If you want to learn more about how to monitor your jobs, please refer to the Automic Automation product documentation at Monitoring Tasks.

This page includes the following:

Monitoring the S3 Job Details

Go to the list of tasks in the Process Monitoring perspective.
Find the S3 task.
Select the task and click the Details button.

The Details pane opens on the right hand side and shows a summary of the execution of the selected task. The General and Job sections of the Details pane display information about the object configuration and its execution. The Object Variables section displays information that the S3 system reports back to Automic Automation for the following jobs:
- Copy File Job returns the number and list of files successfully handled by the job:
  - &NUMBER_OF_COPIED_FILES
  - &LIST_OF_COPIED_FILES
  - &LIST_OF_OVERWRITTEN_FILES
- Delete File Job returns the number and list of files successfully handled by the job:
  - &NUMBER_OF_DELETED_FILES
  - &LIST_OF_DELETED_FILES
- Download File Job returns the number and list of files successfully handled by the job:
  - &NUMBER_OF_DOWNLOADED_FILES
  - &LIST_OF_DOWNLOADED_FILES
- Exist File Job returns:
  - &FILE_EXIST: The variable value is either true or false, depending on if the file exists in the S3 bucket or not.
  - &FILES_COUNT: The variable value is the number of files retrieved by the job. It is available only if the Use Regex option is selected in the job configuration.
- Monitor File Job returns the &FILE_URI variable. The variable value is the URI pointing to the respective file in the S3 bucket.
  
  AWS Example
  
  https://source-test-bucket1.s3.ap-south-1.amazonaws.com/testFile.txt
  
  This variable only has a value if you have selected the Return File URI checkbox on the Monitor File Job definition. Otherwise, it remains empty.
- Move File Job returns the number and list of files successfully handled by the job:
  - &NUMBER_OF_COPIED_FILES
  - &LIST_OF_COPIED_FILES
  - &NUMBER_OF_DELETED_FILES
  - &LIST_OF_DELETED_FILES
  - &LIST_OF_OVERWRITTEN_FILES
- Upload File Job returns the number and list of files successfully handled by the job:
  - &NUMBER_OF_UPLOADED_FILES
  - &LIST_OF_UPLOADED_FILES
  - &LIST_OF_OVERWRITTEN_FILES

Canceling S3 Jobs

You can execute Jobs and cancel the corresponding tasks in the Process Monitoring perspective.

Canceling means that the Automic Automation task is canceled as well as the job on the S3 environment. If for some reason canceling the execution fails - for example, if you do not have the rights to cancel the execution - the execution remains active and that information is visible on the job report.

For more information on how to work with tasks on the Process Monitoring perspective, refer to the Automic Automation documentation at Working with Tasks.

S3 Agent Connection

If the Agent stops working or loses connection, the Job execution stops but resumes as soon as the Agent is connected again.

S3 Reports

When jobs are executed, Automic Automation generates output files and reports that you can open from the user interface. Combined with the Execution Data, these reports provide comprehensive information about the status and the history of your executions. This information tracks all processes and allows you to control and monitor tasks, ensuring full auditing capability.

Reports are kept in the system until they are explicitly removed by a reorganization run. During a reorganization run, the Execution Data can be kept. Keeping the execution data is an advantage, because reports can consume a large amount of hard drive space. Removing the reports from your database does not mean loss of important historical data.

The following reports provide a comprehensive insight into the executions of your jobs:

Agent log (PLOG)

This report captures everything that has to do with the Job execution from the Automic Automation perspective: When the Job started, the Job configuration parameters, whether there has been any validation and its results, and so on. It also contains the response received from the target system but in a format that cannot be parsed for further processing.

Since this report logs all that happens during the execution, it is your first place to go if you need to troubleshoot issues, also if those issues have to do with the configuration of the Job. For example, if you use variables to populate the configuration fields, these variables are resolved at runtime. This report show the actual variable values and not their names.

Report (REP)

This report captures the information that the target system sends to Automic Automation when the Job is executing/has executed. It contains all relevant information about the Job execution and its results. It provides this information in a format that can be easily parsed for further processing (JSON, XML and so on).

Note:

The Report (REP) cannot be parsed if the job throws and exception. In this case, only the exception message is written into both the Report (REP) and Agent (PLOG) logs.

The information is displayed in JSON format following the pattern below:

Copy File Job:

{
    "regex" : "example.*"
    "status" : "success"
   "Item": [
       {
            "sourcefile": "example.txt",
           "destination": "TESTING/example.txt"
           
       },
        {
            "sourcefile": "example123.txt",
           "destination": "TESTING/example123.txt"
           
       }
   ]
}

Delete File Job:

{
    "regex" : "example.*"
    "status" : "success"
   "Item": [
       {
            "sourcefile": "example.txt"
           
       },
        {
            "sourcefile": "example123.txt"
           
       }
   ]
}

Download File Job:

{
    "regex" : "example.*"
    "status" : "success"
   "Item": [
       {
            "sourcefile": "example.txt",
           "destination": "c:/temp/example.txt"
           
       },
        {
            "sourcefile": "example123.txt",
           "destination": "c:/temp/example123.txt"
           
       }
   ]
}

Move File Job:

{
    "regex" : "example.*"
    "status" : "success"
   "Item": [
       {
            "sourcefile": "example.txt",
           "destination": "TESTING/example.txt"
           
       },
        {
            "sourcefile": "example123.txt",
           "destination": "TESTING/example123.txt"
           
       }
   ]
}

Upload File Job:

{
    "regex" : "example.*"
    "status" : "success"
   "Item": [
       {
            "sourcefile": "c:/temp/example.txt",
           "destination": "TESTING/example.txt"
           
       },
        {
            "sourcefile": "c:/temp/example123.txt",
           "destination": "TESTING/example123.txt"
           
       }
   ]
}

You will most probably include the Jobs in other Automic Automation objects such as Workflows or Schedules to orchestrate operations. When you do so, you can use the content of this report on the Post Process pages of the Job definition to trigger further actions based on that content. For more information see, Process Pages in the Automic Automation product documentation.

Directory

This report provides the list of job output files that the job has created for this particular execution and that are available for downloading.

To Access the Reports

Do one of the following:

From the Process Assembly perspective
- After executing a Job, a notification with a link to the report is displayed. Click it to open the Last Report page.
- Right-click a job and select Monitoring > Last Report.
In either case, the most recent report that is created for the object opens up.
From the Process Monitoring perspective
- Right-click a task and select Open Report.
  
  The report of that particular execution opens up. Its runID is displayed in the title bar to help you identify it.
- In the list of Executions for a particular task, right-click and select Open Report.