Advanced Agent Options - Tracing and Workload Distribution

The advanced Agent options allow you to specify the Trace settings, to distribute workload among Agents, to restrict the number of report blocks to be stored in the Automation Engine database and to specify the time interval for the periodic Automation Engine check.

This topic provides information on the following:

Setting the Trace Options
Workload Balancing - Distributing Workload among Agents
- Prerequisites
- Special Case - Maximum Number of Tasks in Parallel
Agent Settings

The options that are available in the Trace Flags section depend on the type of Agent. For example, Windows Agents have an additional section (Additional Trace Options, see screenshot below) that other Agents do not have.

Setting the Trace Options

All agents can write trace files. Like log files, trace files keep a record of activities on a component. However, whereas log files record key events and messages, trace files record the entire program and data flow. This detailed information can be helpful for diagnosing difficult error situations or critical performance bottlenecks.

Although tracing can be useful, it generates huge amounts of data in a short time and this can decrease system performance significantly. For this reason tracing is usually disabled. However, when you have a complex error situation that might be related to one of your agents, Automic Support might ask you to temporarily enable tracing on the agent.

Because tracing can have a noticeable negative effect on system performance, please set trace options only when Automic Support instructs you to do so explicitly. Your support case assistant knows which values to set to collect targeted data with the lowest impact on your system performance.

Trace Flags Section - What Is Traced

In the Trace Flags section you set the trace level for a number of Agent activities (TCP/IP, File Transfers, Events and so on); which options are available depend on the type of Agent. Generally, the trace values are 0–9 to indicate the level of detail collected, but values can be different for some processes. Tracing is disabled when the trace value is 0 (zero).

Rapid Automation (RA) agents have an additional "RA" trace option. When tracing RA agents, you usually specify the highest trace level (9) but, as with all trace options, ask Automic Support what they recommend. Please note that although some RA agents, such as the Web Service agent, allow you to set additional third-party library tracing in their UCXJCITX.INI file, you cannot enable third-party tracing remotely. See Agent RA Core.

Additional Trace Options - How Trace Data Is Collected

Error situations that occur sporadically or are not reproducible are difficult to trace. If you trace everything in the hopes of catching the error, trace files can quickly become hard to read or too big for the available disk space. Using a combination of the following trace options, you can better target your tracing while managing your disk capacity.

Option	Description
Memory Trace	If an agent allows tracing in memory, you can enable this option. You use it in combination with the Trace Search Key. The trace messages will be saved to a memory buffer until the search string is found, at which time tracing stops and the messages are written to one trace file on the disk. Without this option, trace files are written directly to the disk, as one or more files, depending on the Max. Trace Size setting.
Memory Buffer Size	The maximum trace size influences the size of the trace files that are written to the disk of the agent's host. How it works depends on whether the Trace Memory option is enabled: Trace Memory enabled In this case, the trace component reserves the specified space on the disk in preparation for when the trace files is written. If the trace file size in the memory buffer exceeds the reserved space defined here, the trace component will continuously truncate the oldest messages in the trace file to make room for the newest. Trace Memory disabled or unavailable When a trace file reaches the defined limit, the tracing component finishes writing the message or command that it was writing at that time. When the size of the current trace file exceeds this limit, the file is closed and a new trace file is opened. In all cases, the real size of the trace file tends to be larger than the maximum size defined because the trace component finishes writing the message or code that it is logging when the size limit is reached. Some script statements can be extremely long, so the file size will be noticeable larger than the limit set. Important to understand is that the purpose of the Max. Trace Size is to limit the space used in memory. The file representation that is then saved on the disk can be much bigger. For example, if you are using memory tracing, the file will be much bigger, because the trace data is written to the memory buffer in binary form but when the buffer is flushed onto the disk, the trace file is written in hexadecimal form, which explodes the file size. If you set the Max. Trace Size to 0 (zero) without memory tracing, this means that the trace file size is unlimited. However, with memory tracing set, the trace size cannot be lower than the default of 32MB. So, if you enter a value less than that, the system automatically treats it as if it were 32MB. Regardless of the trace settings, if the agent crashes then tracing also stops. The last record will not be written. When the agent starts again, all trace options will be reset to the defaults that are defined in the agent's INI file.
Trace Search Key	You can enter a text string, such as a specific error message, that signals the end of tracing. When the tracing component encounters this string, it finishes writing the current message and then it stops tracing and resets all trace options to "0" (zero). You use this to help you focus on the area where the problem occurred and to manage disk space by preventing excessive tracing.

Option

Description

Memory Trace

If an agent allows tracing in memory, you can enable this option. You use it in combination with the Trace Search Key. The trace messages will be saved to a memory buffer until the search string is found, at which time tracing stops and the messages are written to one trace file on the disk. Without this option, trace files are written directly to the disk, as one or more files, depending on the Max. Trace Size setting.

Memory Buffer Size

The maximum trace size influences the size of the trace files that are written to the disk of the agent's host. How it works depends on whether the Trace Memory option is enabled:

Trace Memory enabled

In this case, the trace component reserves the specified space on the disk in preparation for when the trace files is written. If the trace file size in the memory buffer exceeds the reserved space defined here, the trace component will continuously truncate the oldest messages in the trace file to make room for the newest.
Trace Memory disabled or unavailable

When a trace file reaches the defined limit, the tracing component finishes writing the message or command that it was writing at that time. When the size of the current trace file exceeds this limit, the file is closed and a new trace file is opened.

In all cases, the real size of the trace file tends to be larger than the maximum size defined because the trace component finishes writing the message or code that it is logging when the size limit is reached. Some script statements can be extremely long, so the file size will be noticeable larger than the limit set.

Important to understand is that the purpose of the Max. Trace Size is to limit the space used in memory. The file representation that is then saved on the disk can be much bigger. For example, if you are using memory tracing, the file will be much bigger, because the trace data is written to the memory buffer in binary form but when the buffer is flushed onto the disk, the trace file is written in hexadecimal form, which explodes the file size.

If you set the Max. Trace Size to 0 (zero) without memory tracing, this means that the trace file size is unlimited. However, with memory tracing set, the trace size cannot be lower than the default of 32MB. So, if you enter a value less than that, the system automatically treats it as if it were 32MB.

Regardless of the trace settings, if the agent crashes then tracing also stops. The last record will not be written. When the agent starts again, all trace options will be reset to the defaults that are defined in the agent's INI file.

Trace Search Key

You can enter a text string, such as a specific error message, that signals the end of tracing. When the tracing component encounters this string, it finishes writing the current message and then it stops tracing and resets all trace options to "0" (zero). You use this to help you focus on the area where the problem occurred and to manage disk space by preventing excessive tracing.

To Start Agent Tracing

To start tracing, you need to set trace options that specify exactly what data is collected. You can only set tracing options for agents that are enabled (running).

Go to the Administration Perspective if you are not already there.
Expand the Agents & Groups menu and select Agents. Upon accessing the Agents list, the filter pane is open by default. This allows you to immediately perform a search using the Agent Name field, thus simplifying the work with the list.
Right-click the agent and select Advanced Options.

To Stop Agent Tracing

Generally, when you set trace options for an agent, you specify a Trace Search Key, a string that signals the end of tracing, so that tracing stops automatically.

Tracing stops in one of the following ways:

You define a text string in the search options (Trace Search Key) that signals the end of the trace, such as a specific error message.
The agent runs into a critical error and stopped. In this case, when the agent starts again, tracing options are reset to the defaults that are defined in the agent's INI file.
You disable tracing by manually changing the trace options to stop tracing:
1. Open the Agents list. Upon accessing the Agents list, the filter pane is open by default. This allows you to immediately perform a search using the Agent Name field, thus simplifying the work with the list.
2. Right-click the agent to select Advanced Options.
3. Set all options in the top part of the dialog to 0 (zero), and then click the Apply button.

Workload Balancing - Distributing Workload among Agents

By default, all jobs and file transfers are processed without any agent limitations. At closer view, it becomes obvious that tasks contain different statements. Some of them are CPU consuming or have long runtimes. The Automation Engine has a resource concept to consider the agent workload during processing, which allows different emphasis to be put on jobs and file transfers. Agents have a specified resource pool and it is possible to define how many resources should be consumed during execution for each job and file transfer.

Automic's resource concept does not specifically refer to CPU time or memory etc. The values specified as resources are abstract values which intend to provide a high level of flexibility for your configuration.

A task starts as soon as the agent has sufficient resources to execute it. If there are insufficient resources, the task changes to status Waiting for resource and will be executed when resources are available again. If there are several tasks waiting, the Automation Engine Priority is the crucial factor for deciding which one should be processed first.

In the Max. Resource Jobs and Max. Resources File Transfer fields you enter the maximum number of resources the agent will provide for jobs/file transfers, the allowed values being 0 - 100000. The default value is set by the system administrator in the UC_HOSTCHAR_* variable, for jobs in WORKLOAD_MAX_JOB and for file transfers in WORKLOAD_MAX_FT. See UC_HOSTCHAR_DEFAULT - Host Characteristics.

Example

An agent has 100 available resources for job executions. Three jobs should be started.

Job A requires 10 resources and can be executed.
Job B requires 50 resources and can also be started. 60 of the available 100 resources are now used and 40 are still available.
Job C requires 50 resources but cannot be executed because there are not enough resources. It switches to a waiting condition, namely Waiting for resource.

Tasks that are in the waiting condition can be started at any time either from the Process Monitoring perspective, right-clicking it and selecting Ignore Agent Resource from the context menu, or via the MODIFY_UC_OBJECT script function. The agent's current resource value is ignored. Task resources are consumed and the agent's maximum resource value is exceeded in the short term.

Take into account the following considerations:

Consumed resources are released as soon as the task has ended.
An agent's resources are valid for all clients, because the same agent can be used in several clients.
Resource consumption can also be allocated for jobs which are started by a Remote Task Manager.
In file transfers, source and destination agent must have sufficient resources as otherwise the file transfer cannot be executed.

Prerequisites

Because agents do not check the resource settings of jobs and file transfers, you must specify some options if you intend to use the Automic's resource concept.

Determine how many resources the agents should provide:

Log on to system client 0000.
Open the UC_HOSTCHAR_DEFAULT variable (see UC_HOSTCHAR_DEFAULT - Host Characteristics) or the other agent characteristics of your agents if you use different variables.
Enter the resources the agent should provide in the keys WORKLOAD_MAX_FT and WORKLOAD_MAX_JOB. -1 means that the resource setting of jobs and file transfers is ignored.

Resource values can be changed on a temporary basis. This value applies until the agent connection or the agent itself is ended.

The default value 1 applies for the resource consumption of jobs and file transfers. This value can be changed as follows:

Open the UC_SYSTEM_SETTINGS variable (see UC_SYSTEM_SETTINGS - System-Wide Settings).
Modify the values for the WORKLOAD_DEFAULT_FT and WORKLOAD_DEFAULT_JOB keys if a different default value should be specified.

The resource consumption of jobs and file transfers that should not use the default value can be specified in the particular object definition. If value "0" is entered in this field, the default value of the variable UC_SYSTEM_SETTINGS is used.

Special Case - Maximum Number of Tasks in Parallel

The Automation Engine resource concept can also be used to limit the number of tasks that can run in parallel on an agent.

Ensure that all jobs and file transfers consume resources of value 1.; you specify this in the UC_SYSTEM_SETTINGS variable. The variable default value is only used if job and file transfer objects have the value 0 specified as resource consumption. Now enter the maximum number of parallel tasks as the resource value in the variable(s) for host characteristics.

Each job and file transfer consumes one resource during its execution. Thus, the agent's resource value acts as the maximum number of tasks running parallel.

Agent Settings

In this section you restrict the number of report blocks to be stored in the Automation Engine database. You can enter a value between 1 - 10000 in the Max. Report Blocks field. The default value is set by the system administrator in the UC_HOSTCHAR_* variable, key MAX_REPORT_SIZE.

On the other hand, you also specify the time interval for the periodic Automation Engine check. Again, the default value is set by the system administrator in the UC_HOSTCHAR_* variable, key JOB_CHECKINTERVAL. The allowed range is 1 - 32767.

See UC_HOSTCHAR_DEFAULT - Host Characteristics.