Centralized Agent Upgrade Process
As of version 12 of the Automation Engine it is possible to upgrade all CAU-supported agents to a new version with a centralized and easy to use automatic procedure.
In this document you find the background information, requirements and steps you have to follow.
Our consultants are experts in upgrading AE systems. Contact Automic, our experts will be pleased to assist you whenever it is necessary.
In order to print the whole contents of the collapsed pages below, use the button Expand all/Collapse all in the menu bar above.
Centralized Agent Upgrade - Video
The Centralized Agent Upgrade is very secure and easy to do, watch our video here:
https://youtu.be/896wvguA9B8
General Information
Concept
The Centralized Agent Upgrade (CAU) as of AWA v12 is another automatic upgrade solution:
- The CAU provides an easy and automatic procedure to upgrade agents to a newer version.
- The agents can run as either the old or new version agent during upgrade.
- There is a switch between agent versions at one point of the CAU, therefore a 'zero downtime' is not possible.
Procedure Summary
- Specify the set of agents you want to upgrade and select the desired target agent version.
- Subsequently the Automation Engine triggers all agents to perform the upgrade.
- Each triggered agent loads all needed upgrade resources from the Automation Engine that are kept in Storage objects, located in client 0000.
- Each agent replaces the existing binaries with the new ones.
- After the binaries have been replaced successfully, each agent shuts itself down providing a specific return code.
- The corresponding ServiceManager restarts the agent(s) based on this return code.
- Each upgraded agent starts up and reconnects to AE again with its new agent target version.
You have to import the new version binaries into the system client 0000. (The binaries are attached to and delivered with Storage objects).
The CAU itself can be carried out from any client of your system, by any user with the necessary authorizations.
Please refer to the list of CAU enabled agents below to check, if a particular agent or agent can be upgraded by using the CAU.
In order for the CAU to work properly you also have to first manually upgrade your ServiceManager to v12 .
Upgrade Preparation
As of v12 the following agents can be upgraded by using the CAU:
Agent type | Agent |
---|---|
OS Agents |
Windows UNIX:
|
Dos and Donts
Make sure that all binaries attached to Storage objects and thus imported into the system are from trusted sources.
Users that have the privilege to conduct a CAU are potentially able to import malicious files as well.
Use the ServiceManager to start agents for the CAU. That way you make sure that the agents after upgrading will restart automatically.
Using a different method to start agents is possible, but you will have to ensure the restart another way as well, because the agents will not restart automatically.
The Storage objects mentioned above will be delivered via Automic Plugin Manager packages.
Automic strongly recommends you use separate binaries for agents, not shared ones. Otherwise errors may occur.
Do not change the names of these objects. The specific name pattern is used to identify the resources' types and versions.
If the names of these objects do not match the pattern, the agent version is not displayed in the update list.
Do not change the resource content.
It is released by Automic and shouldn't be altered.
Take care of the following:
- Automation Engine Installation
- ServiceManager Version
- Initial Agent Installation
- There is no agent installed. This constitutes not an upgrade, but the very first installation.
- There is already an agent installed but it doesn't support CAU.
The upgrade to v12 has to be done manually, before CAU can be used to upgrade automatically to subsequent versions. - You want to do a manual installation for any reasons.
Automic strongly recommends not to use manual and automated CAU steps at the same time, as errors may occur. - Agent upgrade via CAU - Storage Objects
- The corresponding Storage objects are already available in the Automation Engine:
Storage object(s) that contain(s) resources of the desired target version, imported to client 0000.
After a successful import of such a Storage object this new target version is instantly available for the CAU. - A Storage object that contains resources of the current version.
This is necessary in order to be able to revert to the previous version (in effect an 'upgrade' to that version). - Set User Privilege
- Install the Automic Plugin Manager
- Install the downloaded Packages using the Packs page
The Automation Engine installation has to be installed and / or upgraded (ZDU) in the usual way.
Make sure that you upgrade your ServiceManager to v12 manually first.
The CAU is possible as of v12, which applies to the ServiceManager as well.
You have to install an (initial) agent manually in the following cases:
An agent has already been installed that supports CAU functionality (see table above).
Any administrator / user who shall be able to execute an upgrade, CAU or ZDU, needs the privilege Execute system upgrades.
In order to be able to import the resource packages mentioned above you need to install the Automic Plugin Manager. It is itself a plugin and would be part of your Automic Web Interface installation.
Details you find in: Installing and Configuring the Plugin
Install the packages (packs) you downloaded using the Packs page in your AWI installation. For details refer to: Packs Page
Upgrade
Summary of Steps to Take
As a system administrator working in client 0000 you have to follow these steps:
- Download the PM package for the CAU for the desired agent version you want to upgrade to.
- Import the PM package into client 0000.
- Filter and multi-select all agents to be upgraded in the Agents page of the Administration perspective.
- Select the context menu item Upgrade for these agents and specify the desired agent target version.
- Look up the CAU related statistics / reports for details if needed, in the Upgrade History page.
- Refresh the Agents page to check:
Successfully upgraded agents will be listed with their new version.
Detailed Upgrade Steps
I. and II. Download and Import Plugin Manager Packages for CAU
The necessary agent resources for the CAU are kept in storage objects (AE object type = STORE).
Therefore you have to make these objects available in the AE before performing CAU actions.
The delivery of these objects is realized via Automic Plugin Manager packages.
These packages are available for each released agent version on the Automic Marketplace.
In case a reversion of the CAU is a possibility, make sure to also import packages for the version you are upgrading from.
To import PM packages for CAU follow these steps:
- Download the Plugin Manager package for CAU for the desired target agent version from the Automic Marketplace.
- Logon to AWI in client 0000.
- Navigate to the Packs view in the Administration perspective.
- Import the Plugin Manager package downloaded previously.
- Click on Replace, if needed.
The old version will not be overwritten by Replace, the new agent version will just be added to the already available ones. - After the Plugin Manager package was successfully imported, the new agent target version is available for the CAU.
Important Notes:
Removing a Plugin Manager package via Plugin Manager plugin will not remove already available agent target versions for CAU.
The versions of all imported packages are accumulated in the AE.
Do not change the names and / or content of Plugin Manager packages or related Storage objects.
If the agent is busy at the time the Automation Engine tries to trigger the upgrade, the agent blocks the upgrade until it is idle again (for example Job objects, script functions or file transfers).
III. and IV. Select Agents to be Upgraded and Specify Version
Please follow these steps:
- Log in to any client of the system using AWI. (You need the respective permissions to display the Administration perspective.)
- Navigate to the Administration perspective.
- Click Agents & Groups, then click Agents. In the Agents page select one or more agents you want to upgrade to the new version.
From the context menu, select Agent Upgrade. - In the following dialog, if the import of Plugin Manager packages was successful, you can find a list of the new versions of agents you can upgrade to:
In this image an SQL agent has been selected for upgrade and therefore is listed, but since it is not yet supported by the CAU, a manual upgrade is required.
The table columns display the following information: - If desired, enter a custom upgrade text in the field below the list. This text will be part of the related CAU statistic record name.
- Select the appropriate option, Agents with ServieManager Link Only or All Selected Agents.
Difference:
The option Agents with ServieManager Link Only starts the upgrade only for agents that have been started via ServiceManager.
The option All Selected Agents starts the upgrade of all agents you selected in the previous dialog, if they are CAU compatible, regardless of a link to the ServiceManager.
If none of the selected agents is linked to the ServiceManager, the option Agents with ServieManager Link Only will be hidden. - Click Upgrade, if you are ready to perform the upgrade.
Afterward a dialog will pop up showing a message regarding the upgrade status for all selected agents per platform:
This dialog is just an information about the start of the upgrade.
For detailed reports refer to the Upgrade History page (see steps V. and VI. below).
You can also filter for particular agents, using the button Filter in the toolbar at the right hand side of the Agents page.
Column | Description |
---|---|
Number of Agents | Number of all agents available for the platform in the second column. |
Platform | The respective platform of the agents you selected in the Agents page. |
Target Version |
The target versions available for that agent type, ordered in descending order, so the latest version is shown at the top. If there is more than one version available for that platform, a drop down arrow at the end of the row indicates that you can select the appropriate version there. |
V. and VI. Check Reports and Upgrade History
Immediately after the upgrade has been started, you can look up any detailed information and reports in the Upgrade History page.
- Still in the Administration perspective, select the Upgrade History page in Agents & Groups.
This page will be empty, displaying a message to that effect, if no upgrades have been started or run yet. - In the Upgrade History page you can see a list of all agents that are currently being upgraded or whose upgrades have recently finished.
In addition their status is displayed, for example Waiting for host. This means, the agent is busy so the upgrade has been postponed to wait for the agent to become available.
Example of the Upgrade History page display:
Initially only the upgrade runs as a whole are visible.
Use the arrow icon at the left of each row to expand the display to the child/children (agent or agents individual deployment) of an upgrade run.
Table details: See below. - Use the context menu of any of the displayed agents and its option Open Report, which in turn will open the detailed report of that agent's upgrade.
The Upgrade History table shows the following information:
Column | Description |
---|---|
Name | Name of the CAU run, can be configured in the Custom Upgrade Text field when you select the target version. |
Status | Status of the run - or the child, which would be an agent or agents individual deployment, depending on the selection you made when choosing the agents to be upgraded. |
RundID | RunID of the CAU run or the individual agent deployment. |
Parent | (only filled, if child is displayed) Shows the CAU run which is parent of an individual agent's deployment. |
User | User name of the user executing the CAU. |
Agent | (only filled, if child is displayed) Agent name of the individual agent being upgraded. |
Target Version | New version to be upgraded to. |
Start Time/End Time | Start and end time of the run or the individual agent upgrade. |
Runtime | The upgrade's duration. |
You can configure the necessary AE system modules to execute objects on the CAU's completion automatically.
You have to be logged in into system client 0000 to be able to configure the necessary variables and keys.
Execute on Upgrade - Configuration
In order to configure the automatic execution of objects after the CAU has been completed, the system variables UC_HOSTCHAR_DEFAULT - Host Characteristics and - for a particular agent - the UC_EX_HOSTCHAR - Assigning Agents to Host Characteristics variable have to be edited or created.
You first have to create the object or objects to be executed after the upgrade has been finished.
Afterward you set the values in the respective UC_HOSTCHAR_DEFAULT key, following these steps:
- In the Process Assembly perspective find the HOST_VARIABLES folder. The system variable UC_HOSTCHAR_DEFAULT is located inside it. Double click the variable to open and edit it.
- In the Variable page you will see a list of keys you can edit.
Find the keys EXECUTE_ON_DEPLOYMENT_RUN and EXECUTE _ON_DEPLOYMENT_AGENT.- If you want to execute an object of your choice after the entire deployment run has been finished, use EXECUTE_ON_DEPLOYMENT_RUN.
- If you want to execute an object of your choice after a particular agent upgrade has been finished, use EXECUTE_ON_DEPLOYMENT_AGENT.
- This step applies to both keys:
On the right hand side in the table use the column Value 1 to enter the name of the object that should be executed after the upgrade.
In case you want to execute more than one object, use the object defined here in the necessary Automation Engine setup to trigger consecutive object executions. - Save the system variable.
In order to read or print the report contents of CAU runs, you can use script functions and available script variables. Find examples here:
Using Scripts and Script Variables
Available Script Variables
There are script variables available that you can use in scripts in the Process page of an object you referenced to read the respective values, namely:
Use with EXECUTE_ON_DEPLOYMENT_RUN, so the variables listed will return the values as described here:
- &UC_EX_CAU_NAME# - Custom text of agent upgrade
- &UC_EX_CAU_RUNID# - Top statistics RunId of deployment
- &UC_EX_CAU_STATUS# - Status of whole CAU run (it can be ENDED_OK or ENDED_NOT_OK)
Use with EXECUTE_ON_DEPLOYMENT_AGENT, which will return the following values:
- &UC_EX_HOST# - Agent / host name. (For compatibility reasons the variable name without "#" works as well here.)
- &UC_EX_CAU_NAME# - Custom text of an agent upgrade run.
- &UC_EX_DEPLOYMENT_RUNID# - RunId of a child / agent upgrade
- &UC_EX_DEPLOYMENT_STATUS# - Status (number) of child / agent upgrade
- &UC_EX_CAU_RUNID# - Top statistics RunId of deployment
Script Examples:
This script reads all variables from the read buffer and uses / prints them (single agent deployment):
:READ &UC_EX_CAU_NAME#,,
:READ &UC_EX_CAU_RUNID#,,
:READ &UC_EX_HOST#,,
:READ &UC_EX_DEPLOYMENT_RUNID#,,
:READ &UC_EX_DEPLOYMENT_STATUS#,,
:P "CAU RUN INFORMATION:"
:P "CAU custom text=",&UC_EX_CAU_NAME#
:P "CAU runID=",&UC_EX_CAU_RUNID#
:P "AGENT DEPLOYMENT INFORMATION:"
:P "Agent=",&UC_EX_HOST#
:P "Agent deployment runID=",&UC_EX_DEPLOYMENT_RUNID#
:P "Agent deployment status=",&UC_EX_DEPLOYMENT_STATUS#
Use PREP_PROCESS and this example script for a CAU report (an entire CAU run):
For a whole CAU run use "CAU" as parameter report type in PREP_PROCESS_REPORT.
:SET &HND#=PREP_PROCESS_REPORT(,&UC_EX_DEPLOYMENT_RUNID,"CAU")
:PROCESS &HND#
:SET &RET#=GET_PROCESS_LINE(&HND#)
:PRINT &RET#
:ENDPROCESS
Use PREP_PROCESS and this example script for a "REP" report (single agent deployment):
For processing the detailed report of a single agent upgrade use "REP" as parameter report type in PREP_PROCESS_REPORT.
:SET &HND#=PREP_PROCESS_REPORT(,&UC_EX_DEPLOYMENT_RUNID#,"REP")
:PROCESS &HND#
: SET &LINE#=GET_PROCESS_LINE(&HND#)
: PRINT &LINE#
:ENDPROCESS
- After the agent upgrade has run, the Automation Engine triggers a cleanup of all binaries that are now not needed anymore.
- Basically what happens is this: The agent that has been upgraded re-connects to the AE. If its target version agrees with the CAU record of the AE, the AE sends a cleanup message to the agent.
- The agent then removes the binary file of the previous version from the system.
Because of this cleanup function which will be triggered on the agent's re-connect, it is necessary that you provide the binaries as CAU packages of previous versions in case you want to revert the CAU. Take care to import them should they not be present in the system yet.
Skip Upgrade
It is possible to skip the upgrade of agents that you initially registered for an upgrade, if they are in the status Waiting for host.
Two possibilities exist to skip the upgrade:
- Use the web interface (see description below).
- Create a script using the Script function MODIFY_SYSTEM.
Upgrade Skip in AWI:
- In the web interface, navigate to the Administration perspective, as also described above and open the Upgrade History page.
- Find the agents which are in the status Waiting for host, choose one or more whose upgrade you want to skip.
- Right-click a selected agent or agents (multi-select is possible, using the STRG or SHIFT key) and from the context menu, choose Skip upgrade:
Revert to Previous Version
How to Revert to a Previous Version
You may want to revert the agent's upgrade in case your preferences or system occurrences make it seem appropriate.
This function is technically an upgrade to another agent version.
Therefore you just have to follow the steps described above for the CAU itself.
What you have to do
- Import the corresponding packages into the system to make them available for reverting to a previous version.
- In step IV. 4. of the CAU description above choose the agent version you want to revert to and finish the upgrade procedure as described.
- To check the details of the previous version, open the Upgrade History page. Follow the steps described above in V. and VI. Check Reports and Upgrade History.
Background Information
System Behavior during a Centralized Agent Upgrade (CAU)
- When you select the Latest option for CAU, the latest upgrade resources available in AE are applied. If the agent runs already in an even newer version it is downgraded to the latest version available in the AE
- The CAU can be used to downgrade an agent as well. From a technical point of view it is an upgrade to a previous version, no source / target version checks are performed to prevent a downgrade.
- The CAU can be triggered for agents that are not connected to the AE, too.
The AE remembers the desired upgrade and triggers it as soon as the agent gets connected to the AE and is available for operation.- Even for agents that have never been connected to AE (no current version info available) the upgrade can be triggered. They will be upgraded when they connect to AE for the very first time.
- Agent connects with wrong target version after upgrade:
If the CAU has been already successfully performed for an agent and the agent connects to the AE with a wrong target version subsequently, the CAU is repeated.- This means, an upgrade is forced to bring the agent to the last target version that was selected in the last CAU action.
- In this case in the Upgrade History of system client 0000 the entries titled "REPAIR" will show up to indicate that the agent was repaired by a forced upgrade to the desired target version.
- When using cluster agents, you may observe this:
Agent 1 that was upgraded successfully is terminated. The redundant agent 2 starts up with the same agent name but the old version.
Consequently agent 2 will be upgraded to the same version as agent 1 - When performing CAU the AE basically triggers all selected agents to perform an upgrade. If an upgrade cannot be triggered immediately the agent remains in the status Waiting for host regarding this upgrade.
- If an agent remains in this status the pending upgrade can be skipped by the user via web interface and / or script function.
- If an agent already performs the actual upgrade (downloading and replacing files, restart,...) the upgrade cannot be skipped anymore.
- An agent cannot be upgraded and run agent jobs at the same time.
- If an agent is busy (e.g. executing a job or a file transfer) the upgrade process is put on hold until the agent is idle again and ready to perform the upgrade.
- While an agent is performing an upgrade it is suspended and tasks to be executed change to the status Waiting for host
- Script functions:
Some server-side script functions request data from agents. These script functions are suspended during the agent upgrade and resumed after the upgrade is completed.
-
There are two different report types available in the CAU context.
- CAU: AE reports triggering an upgrade for each agent, as well as the end status, or a skipped upgrade process.
- DEPLOY: An agents reports all steps during an upgrade such as downloading / upgrading binary resources as well as it's final termination for restart.