Updating the Rule Engine

As a Flink administrator this section will guide you through the Rule Engine upgrade process.

This page includes the following:

Upgrade Process for v12.2.0 and Above

The following steps are advised when migrating from any old version of Apache Flink to a new version. This upgrade process lets keep the state of the rules.

Notes:

  • The term old refers to your current installation of Apache Flink cluster used by the IA agent. The term new refers to the target Apache Flink cluster.

  • On Windows, use flink.bat

Important! This update only applies when a stateful rule (i.e. the State Change Rule) is running on an existing Apache Flink cluster.

To upgrade the Rule Engine (Apache Flink)

  1. Stop the IA Agent
  2. Connect to the old Apache Flink Job Manager machine and navigate to the Apache Flink bin directory
  3. Using the command-line interface, identify running jobs you want to migrate by entering either command:

    ./flink list - List scheduled and running jobs (including their JobIDs)
    ./flink list -r -List running jobs (including their JobIDs)

  4. Extract the run ids of each running jobs from the job name to build the savepoint.properties file
    The run id can be extracted from the list of flink jobs where their names follow the pattern:

    <namespace>|<agent-name>|<client>|<run-id>

  5. Cancel each job in the old cluster alongside with creating corresponding snapshots.
    See: Triggering Savepoints.

    To easily restore the savepoints later on, it is recommended to have them created inside a sub-directory having the name of the Apache Flink job id:

    ./flink cancel -s /tmp/<flink_job_id> <flink_job_id>

  6. Copy the savepoints folder(s) to the new cluster
  7. Create a file savepoints.properties with the following content:

    <run-id>=<savepoint-path>

    The savepoint path is the corresponding savepoint job path i.e. /tmp/bcacb28fcb99d71e1c0d9641b28921f6/savepoint-bcacb2-6503a8eac076, on the new cluster.

  8. Upgrade the rule package to a version supporting the new cluster
  9. Reconfigure the IA Agent to use the new cluster
  10. Copy the savepoints.properties file to the working directory of the IA Agent
  11. Start the IA Agent
  12. Verify that all jobs have resumed, then delete the savepoint.properties file

Upgrade Process for v12.1.0 and Above

Important! Preserving the rule state from one version of Flink to the next is not supported in v12.1.

Stateless Upgrade with ONE Installer

  1. Stop the IA Agent
  2. Stop the Rule Engine in Service Manager
  3. Update all rule packages to the version which supports the new Rule Engine version
  4. Reconfigure the Service Manager to the new rule engine by changing the start-up path and working directory
  5. Start the Rule Engine in Service Manager
  6. Start the IA Agent
  7. The Rules will be restarted on the new Flink cluster

Stateless Upgrade with External Cluster

  1. Stop IA Agent
  2. Update all rule packages to the version which supports the new Rule Engine version
  3. Reconfigure flink.host and flink.web_port of IA Agent to zpoint to new Flink cluster
  4. Start IA Agent
  5. The Rules will be restarted on the new Flink cluster
  6. Cancel each job in the old cluster alongside with creating corresponding snapshots.
    See: Triggering Savepoints.