Upgrade Process

The Zero Downtime Upgrade allows you to upgrade your AE system with no downtime. The System Upgrade wizard in the Automic Web Interface guides you through each step of the upgrade process. There is no need for scripting or for using AE internal components.

You can also use the Zero Downtime Upgrade when the alternative MS SQL Server schema is already in use. However, you cannot use it to switch to this schema. For more information, see Integrating an Alternative MS SQL Server Schema.

The communication between the AE and AWI as well as between the AE and the Windows, UNIX and Java Agents uses TLS/SSL through a secure WebSocket (WSS). These components establish a connection with the Java communication process (JCP).

The Java communication process (JCP) uses trusted certificates to prove its identity to other communication partners and requires a keystore to work with these certificates. Therefore, you have to install and configure the Java communication process (JCP) and make sure you have the required certificates.

The Windows, UNIX and Java agents can use different parameters in their corresponding INI file to define how to handle the certificates.

Note: The TLS/SSL implementation does not apply to the HP-UX and 32 bit ZLINUX Agents, as they are no longer supported.

More information:

Important!

Only use the ZDU to upgrade systems that were installed manually or test systems that were installed using the ONE Installer. You cannot migrate to a cointainer-based system nor upgrade a container-based system using the Zero Downtime Upgrade.
It is not possible to carry out a Centralized Agent Upgrade (CAU) parallel to a Zero Downtime Upgrade.
The AWI instances in versions higher than 12.3 connect to the new JCP, which must have the required certificates in place.
During ZDU, the new JCP is only available after the connections have been switched in step four of the wizard and the primary work process (PWP) in the new version is running.

Therefore, do not upgrade the AWI instance you are using for the Zero Downtime Upgrade. Make sure you have a separate, upgraded instance of your Automic Web Interface available for after the connections are switched.

If you need help, contact our support team. Our consultants are experts in upgrading AE systems and will be pleased to assist you whenever it is necessary. For more information, see Support.

This page includes the following:

Overview

It is recommended to start an upgrade only if the necessary preparation steps for setup and testing have been taken care of.

Two different modes are activated during the upgrade: the compatibility mode and the parallel mode.

Compatibility mode

This mode is active during the whole upgrade process, even if you carry out a rollback. Certain system optimization functions are not available when in compatibility mode and the system performance is reduced by about a third.

You cannot switch a work process (WP) to a dialog work process (DWP) while in compatibility mode.
Parallel mode

This mode is only active when server processes from the current and new version are active at the same time. Current version work processes (WPs) may slow down until the upgrade is completed.

During the upgrade you will be asked to test extensively if everything works before finalizing the upgrade. If any issues arise, you can use the Rollback option. It is available only in step four (Connections) of the wizard, if it has been processed successfully. For more information, see Rollback.

Note: All database changes persist even if you use the rollback option.

If an Automation Engine process running in the old version stops, it cannot be restarted. If all Automation Engine processes stop (for example, due to power failure), the Zero Downtime Upgrade finishes the upgrade process. In this case, you cannot perform a rollback. After the upgrade process has finished, the old server processes end automatically.

Once you have finished the upgrade process by clicking Next in the final step of the wizard, a rollback is no longer possible.

When the upgrade process finishes, the compatibility mode ends and the system runs normally again.

Important!

It is strongly recommended to carry out the following upgrade steps in the order they are listed in the CAPKI documentation. For more information, see CAPKI - Securing the ServiceManager.
It is possible to set the privilege for this kind of upgrade for any User object, but it is recommended that only the system administrators coordinate and conduct upgrades.
The upgrade must be carried out in the same client from start to finish.
Do not make any changes in Client 0 while using the DB Load Utility during the upgrade process as initial data is inserted/updated. Otherwise, you will have inconsistencies in your system.
Utilities cannot be rolled back once the database has been upgraded in step two (Processes) of the wizard. You have to use the utilities in the new (target) version from then on.
Automation Engine utilities are external applications. Make sure that no utilities run during the Zero Downtime Upgrade process.
The Automic Web Interface and the AE server must have the same release version. To check other compatibility issues, see Compatibility Information or access the compatibility matrix directly.

Prerequisites

Make sure your system meets the system and user prerequisites, otherwise the Zero Downtime Upgrade will not work properly.

System Setup

To properly carry out the Zero Downtime Upgrade, you have to use a distributed installation. These installations can be located either on two separate hosts or on two separate installations on the same host, but in different bin directories. For more information, see ZDU - Distributed Installation.

The upgrade process makes reference to either the current (base) or the new (target) system, which also represent the current and new version in the upgrade scenario.

If you use a proxy and a firewall in the systems used for the upgrade, you have to configure the ports accordingly. For more information, see Configuring Firewall and Ports.

Information security services, such as the TLS/SSL implementation of the ServiceManager, require features provided by the CAPKI package. You need to install CAPKI on all computers in which the AE processes or the ServiceManager and any of its clients will run.

Important! In a UNIX environment, do not overwrite the existing installation. In UNIX this can happen without a warning even if the binaries are currently in use.

The communication between the AE and AWI uses TLS/SSL through a secure WebSocket (WSS). The upgraded AWI instance establishes a connection with the new Java communication process (JCP), which uses trusted certificates to prove its identity to other communication partners and requires a keystore to work with these certificates. Therefore, you have to install and configure the new Java communication process (JCP) and make sure you have the required certificates.

More information:
Set User Privileges

Make sure that the respective user or users (usually system administrators) have the necessary permissions. The user conducting the upgrade must have the privilege Execute system upgrades set in the corresponding User object. For more information, see Granting Automation Engine Privileges.

Upgrade and Rollback

On the Administration perspective, select Automation Engine Management > System Upgrade and click Run System Upgrade Wizard to start the Zero Downtime Upgrade.

While upgrading, you can leave the process at any time and return to it later. The system stores the information of the step where you left off and opens the System Upgrade page at that step automatically as soon as you return.

Rollback

The rollback function allows you to check tasks or processes that might not run quite as expected while testing the new (target) version. It is visible in step four (Connections) of the System Upgrade wizard, as soon as the server processes from the new (target) version have been started and are running. Using this option rolls back the processing to the current (base) version.

Once you have finished the upgrade process by confirming Next in the final (seventh) step of the wizard, a rollback is no longer possible.

Important! When performing a rollback from versions higher than 12.3, keep in mind that the communication between the Automation Engine and the Automic Web Interface is established through the new Java communication process (JCP). This means you have to start the rollback in the target version of the AWI and continue in the current (base) version of the AWI.

Upgrade Steps

Log on to Client 0 in the Automic Web Interface.

If a user has been granted the privilege Execute system upgrades for the first time, that user has to log off, restart and then log on again to Client 0 with those credentials.
On the Administration perspective, select Automation Engine Management > System Upgrade and click Run System Upgrade Wizard to start the upgrade process.

This starts the compatibility mode and enables upgrading the database with the schema and initial data of the new version.

Note: Leave the page open and use the external DB Load Utility to upgrade the database. For more information, see AE DB Load.

The DB Load Utility carries out the following actions:
- Running system check
  
  If the ZERO_DOWNTIME_UPGRADE variable is set to N and the system is running, DB Load ends the loading process with a corresponding message.
- Version Check:
  
  DB Load checks if the correct version for the Zero Downtime Upgrade is installed and used. If the version is not correct, a corresponding message is returned.
- Check and Load new DB schema and initial data
  - If the ZERO_DOWNTIME_UPGRADE variable is set to N and the system is not running, DB Load loads the new DB schema and initial data. Records are written into the appropriate database table for future reference.
  - If the ZERO_DOWNTIME_UPGRADE variable is set to Y and the system is running, DB Load also loads the new DB schema and initial data. Otherwise DB Load aborts and returns the corresponding message.
  - In addition DB Load checks if the server processes of the new version are already active. If so, DB Load ends and returns the corresponding message.

While this step is processed, the AE system loads the new version database schema and initial data. When you have finished upgrading the database with the DB Load Utility, click Next to confirm the step in the wizard. The Automation Engine checks if the initial data for the current and new version are present and the new version data has been loaded correctly.

Install and start the server processes for the new version parallel to the ones already running in the current version. The new processes are not in use yet.

Important!
- When upgrading to versions higher than 12.3, keep in mind that the communication between the Automation Engine and the Automic Web Interface is established through the new Java communication process (JCP), which must have the required certificates in place before starting the new JCP (see TLS/SSL Certificate Considerations). This means that you can start the server process in your Service Manager, but the primary work process (PWP) of the current version does not recognize it. Therefore, the new JCP is not displayed in the Automation Engine Management > Processes and Utilization page of the Administration perspective until the connections have been switched in step four of the wizard and the primary work process (PWP) in the new version is running.
- Ensure that both, CP and JCP are running.
Make sure you have a separate, upgraded instance of your Automic Web Interface available for after the connections are switched in step four (Connections) of the wizard.

Important!
- Do not upgrade the AWI instance you are using for the Zero Downtime Upgrade. Make sure you upgrade a separate instance.
- As the AWI instances in versions higher than 12.3 connect to the new JCP, do not upgrade AWI and immediately try to continue the Zero Downtime Upgrade in the AWI new version. Remember that the new JCP is only available after the connections have been switched in step four of the wizard and the primary work process (PWP) in the new version is running. Also, remember that the JCP must have the required certificates in place.
There are two main issues you must consider before upgrading the Automic Web Interface to version 21:
- The Automation Engine and the Automic Web Interface as of v21 communicate using TLS/SSL. AWI establishes a connection with the new Java communication process (JCP), which uses trusted certificates to prove its identity to other communication partners. Therefore, make sure you have all required certificates in place before upgrading AWI.
  
  More information:
- As of version 21, it is no longer necessary to install an application server before installing the Automic Web Interface because the delivery artifact includes the AWI Jetty Launcher. This gives you two options:
  - you can keep your application server and upgrade AWI as usual
  - you can install a new AWI version with the Jetty Launcher and use your backed up configuration and plugin files to configure AWI. For more information, see Installing the Automic Web Interface.
Upgrading AWI Using an Application Server

Upgrading an existing installation to the new release upgrades the AWI framework and all its plugins at the same time. Use the same steps to install hot fix packages between releases.
1. Stop the Tomcat Service.
  1. Open your Windows Services manager.
  2. Stop the Tomcat service.
2. Back up current files.
  
  You can make a safety copy of your entire AWI installation, or just the configuration files. Even though you can use very little of the previous configurations in the new version, you might find it helpful to have a copy for later reference when you configure the upgrade.
  - To back up your whole AWI installation, go to your Tomcat installation, open the webapps folder and make a backup copy of the entire awi folder.
  - To back up only the configuration files, go to the folder where you have the current AWI version installed, and make a backup copy of the folder webapps/awi/config.
3. Upgrade the application server.
  
  Important! AWI usually runs on Tomcat. If you want to use another application server for the Automic Web Interface, see Installing the Automic Web Interface.
  
  Before upgrading the application server, check the compatibility matrix to see which version is compatible with AWI. You will find the upgrade instructions and other relevant information on the homepage of your application server provider. For more information, see Compatibility Information.
4. Restart the application server.
  
  You will find the relevant information on the homepage of your application server provider.
5. Deploy the new AWI version (WAR file) on your application server.
  - If you are using Tomcat
    
    Copy the awi.war file to the webapps folder of the Tomcat installation. If the Tomcat is running, the awi.war file will be deployed automatically, creating a folder with the same name.
    
    Important! Deleting the awi.war file removes the corresponding subfolder, which also undeploys the AWI.
  - If you are using WebSphere or WebLogic
    
    Refer to the WebSphere or WebLogic official documentation for more information.
6. Configure the new AWI version.
  
  For more information on how to configure your upgraded AWI, see Configuring the Automic Web Interface.
  
  If you made a backup before you started the upgrade, then you can retrieve the following files and settings from the backup files from the previous to the new webapps/awi/config folder:
  - uc4config.xml - Configuring the Connection Between AWI and AE:
    
    You can replace the new file with your backup.
    
    If you have more than one AE system connection defined, you can allow your users to choose their connection at login, or you can specify any of the defined connections as the only valid connection. You defined this in the automationengine.index setting of the configuration.properties file.
    
    Previously, users could not choose a connection and you had to be sure that your one valid connection was defined first in uc4config.xml.
    
    If you are using self-signed certificates, you can use the trustedCertFolder= parameter to define the path to the folder where the trusted certificates are stored. Otherwise, the certificates should be installed in the Java trust store. For more information, see Securing Connections to the AE (TLS/SSL) and TLS/SSL Certificate Considerations.
  - configuration.properties - Configuring Your Local Setup:
  - logback.xml - Configuring Log Levels:
    
    Do not restore your backed up version.
    
    However, if you changed the default setting for the severity level of event messages (DEBUG) for the root element, then you might want to make the same change for AWI.
  - Theme folder:
    
    If you had customized the color or banner logo in your previous environment, then copy the contents of your backed up webapps/awi/config/theme folder to the folder of the same name in your current AWI version.
    
    For more information, see colors.properties - Configuring the Color Scheme.
Upgrading AWI without Using an Application Server

In this case you install a new AWI instance that includes the AWI framework and all its plug‑ins.
1. Back up the current configuration files.
  
  You can make a safety copy of your configuration files. Even though you can use very little of the previous configurations in the new version, you might find it helpful to have a copy for later reference when you configure the upgrade.
  
  To back up your files, go to the folder where you have the current AWI version installed, and make a backup copy of the config folder, for example (Tomcat)
  - <webserver>/webapps/awi/config for the configuration files
2. Install the Automic Web Interface version that includes the AWI Jetty Launcher, see Installing the Automic Web Interface.
3. Configure the new AWI version.
  
  For more information on how to configure your upgraded AWI, see Configuring the Automic Web Interface.
  
  If you made a backup before you started the upgrade, then you can retrieve the following files and settings from the backup files from the previous to the new webapps/awi/config folder:
  - uc4config.xml - Configuring the Connection Between AWI and AE:
    
    You can replace the new file with your backup.
    
    If you have more than one AE system connection defined, you can allow your users to choose their connection at login, or you can specify any of the defined connections as the only valid connection. You defined this in the automationengine.index setting of the configuration.properties file.
    
    Previously, users could not choose a connection and you had to be sure that your one valid connection was defined first in uc4config.xml.
    
    If you are using self-signed certificates, you can use the trustedCertFolder= parameter to define the path to the folder where the trusted certificates are stored. Otherwise, the certificates should be installed in the Java trust store. For more information, see Securing Connections to the AE (TLS/SSL) and TLS/SSL Certificate Considerations.
  - configuration.properties - Configuring Your Local Setup:
  - logback.xml - Configuring Log Levels:
    
    Do not restore your backed up version.
    
    However, if you changed the default setting for the severity level of event messages (DEBUG) for the root element, then you might want to make the same change for AWI.
  - Theme folder:
    
    If you had customized the color or banner logo in your previous environment, then copy the contents of your backed up webapps/awi/config/theme folder to the folder of the same name in your current AWI version.
    
    For more information, see colors.properties - Configuring the Color Scheme.
  If you made a backup of the plug-ins before you started the upgrade, you can copy them into the <jetty>/plugins/autoinstall folder in the new AWI installation. To do so, either compare the ones that were installed by default and copy the remaining ones or copy the complete backed up plug-ins making sure not to replace existing ones.
Once you have an upgraded instance of AWI available for later on, click Next to confirm the step in the wizard and continue to step four (Connections).
In this step (Connections), the server processes of the current and new versions are switched.

Note: When the connections are being switched to the new version, the WP log file might display the message "UCMAIN_R: Unexpected KICK with type 'RLASS' occurred." or another type of KICK message. You can ignore this message while ZDU is in mixed mode, as the issue resolves itself once the ZDU process finishes.

The running tasks finish on the current version processes while new tasks start on the new version processes until all current version processes are idle. Therefore, all users still connected to the AWI instances of the current version must be disconnected. Also, you must disconnect all agents connected with current version processes manually so that they reconnect automatically with the new version processes. Make sure that the agents can be disconnected without interrupting running executions.

Important! Remember that the new JCP is only available after the connections have been switched and the system is running in the new version.

The wizard offers an interactive page with two panes, where you can disconnect users and agents yourself. The buttons for disconnecting users and restarting agents change the side of the pane and show up over the table for new or old version, depending on the function you are using: Upgrade or Rollback.
- To disconnect one or more users at once:
  
  Access the Users list in the Connections pane. Select the user or users you want to disconnect and click Disconnect User. To disconnect all users, click the respective button. In this case, no selection is necessary.
  
  The disconnected users disappear from the list.
- To reconnect on or more agents at once:
  
  Access the Agents list in the Connections pane. Select the agent or agents you want to reconnect and click Reconnect Agent. To reconnect all agents, click the respective button. In this case, no selection is necessary.
  
  The reconnected agents appear either in the current version or the new version list, depending on the version in which you reconnected them.
The system runs now with the initial data and processes from the new version, while the processes of the current version are subsequently phased out.

Important!
- The Rollback option is available at this point (it is not visible in the previous steps). This is the last opportunity to Rollback before finalizing the upgrade process.
- When performing a rollback from versions higher than 12.3, keep in mind that the communication between the Automation Engine and the Automic Web Interface is established through the new Java communication process (JCP). This means you have to start the rollback in the target version of the AWI and continue in the current (base) version of the AWI.
- This parallel mode should last as long as needed for proper testing. Testing the system allows you check if all processes and tasks run smoothly. Once you have tested your system and you have determined that the upgrade was successful, you can carry out the last step of the process to finish the upgrade. For more information, see FAQ - Zero Downtime Upgrade.
Now that the connections have been switched and the system is running in the new version, log off AWI and log on again using your upgraded, new Automic Web Interface instance.

Access the System Upgrade page again. The system stores the information of the step where you left off and opens the wizard at that step automatically as soon as you return. Click Next to confim the step in the wizard and continue.

Also, now you can see the new JCP is now listed in the Automation Engine Management > Processes and Utilization page of the Administration perspective.
If testing was successful, click Next. Analytics, CDA, and other plug-ins and additional components can be upgraded here.
Check the memory entries in the database and delete the ones referring to the old version.
If you want to upgrade your agents, use the Centralized Agent Upgrade (CAU). For more information, see CAU - Upgrade Process.

Clicking Next finishes the upgrade process. This is only possible, if no current version processes are active anymore. This also ends the parallel and compatibility modes. Once you do this, a rollback is no longer possible.

Note: If the error message ZERO_DOWNTIME_UPGRADE - 'FINALIZE' not possible. Work processes to shut down are not idle yet. Please try again later. is displayed, schedule tasks may be active. Wait for the scheduled tasks to change their execution period (usually the next day), then return to the wizard and try executing the option again.

Rollback Steps

Rolling back is only possible in step four (Connections) of the System Upgrade wizard. You can leave the upgrade process at any time and return to it later. The Automation Engine system stores the information of the step where you left off and opens the System Upgrade page at that step automatically as soon as you return.

In step four (Connections) of the wizard, select Rollback and follow the instructions.

Important! When performing a rollback from versions higher than 12.3, keep in mind that the communication between the Automation Engine and the Automic Web Interface is established through the new Java communication process (JCP). This means you have to start the rollback in the target version of the AWI and continue in the current (base) version of the AWI.
The system switches back using server processes from the current version, while the new processes are subsequently phased out.

Exception! The primary role (PWP) remains with a new WP until Restart has been executed.
Two main scenarios may cause a Rollback:
1. Something in AE object definitions or environment settings must be adapted to work correctly with the new version. In this case, make sure you solve the issue and continue with step one (Initial Data) of the wizard.
  
  To return to step one (Initial Data) from step four (Connections), make sure you are in Rollback Mode and click Next.
2. The initial data or the processes of the new version are not working as expected. In this case contact our support department for assistance. For more information, see Support.
  - If new components have to be installed to solve the issue, execute the option Restart in step four of the wizard (Rollback Mode). This ends the parallel mode, switches back the PWP role to the current version and shuts down running components of the new version so that they can be replaced.
    
    Note: When rolling back from parallel mode, the current version terminates the processes from the higher version. As the current version can only terminate processes that it already knows, the user must manually terminate all other processes. For example, during a ZDU from 12.3 to a higher version, the current version (12.3) cannot terminate the JCP.
  - Install components you may have received from our support department and continue with the Upgrade procedure step one, clicking Next to start the check for initial data, as described above.
  Note: If you click Restart and the error message ZERO_DOWNTIME_UPGRADE - 'Restart' not possible. Work processes to shut down are not idle yet. Please try Restart again later. is displayed, schedule tasks may be active. Wait for the scheduled tasks to change their execution period (usually the next day), then return to the wizard and try executing the option again.