Setting up the Streaming Platform

On Windows

Manual setup instructions for the Streaming Platform on Windows are currently not provided.

It is recommended that you use the Linux installation in a production environment.

Note: If you would like to evaluate the Streaming Platform on Windows, use ONE Installer to setup a single-node system.

For more information, see: ONE Installer - Single-Box Installation

On Linux

Follow these steps to set up the Streaming Platform.

Step 1: Download the required zip file from ca.com

Log in to the Support page and click on Download Management .
Using the search bar, enter CA Automic followed by package you want to download.

Example: Automic Automation

After downloading the file, unzip it to a temporary directory.
Locate the kafka tgz file in the following directory: <Automic>/External.Resources/kafka.
Untar the kafka tgz file. We will copy this file to the target installation machine in Step 2 below.

Step 2: Start the Streaming Platform

On the target installation machine, create a new directory that looks like this: <automic>/External.Resources/kafka.
Copy the untarred kafka tgz file content into: <automic>/External.Resources/kafka directory. The directory should look like this:

bin

config

libs

site-docs

NOTICE

Copy the following script into Notepad (or a text editor) and save it as start.sh in the following directory: <Automic>/External.Resources/kafka/bin.

Important!

Note that in the following script you must replace the %JDK% variable with your actual jdk path.
Replace %ZOOKEEPER_CONNECT% variable with the actual [host]:[port] of Zookeeper (for example the default is localhost:2181).

#!/usr/bin/env bash
echo Starting up Zookeeper

stop_kafka() {
    echo Stopping kafka > kafka_stop.out
    ./kafka-server-stop.sh
    echo Stopping zookeeper >> kafka_stop.out
    ./zookeeper-server-stop.sh
    echo Shutdown finished >> kafka_stop.out
    exit
}

trap stop_kafka SIGHUP SIGINT SIGTERM SIGUSR1 SIGKILL

export JAVA_HOME=%JDK%
export PATH=${JAVA_HOME}/bin:$PATH

./zookeeper-server-start.sh ../config/zookeeper.properties > zookeeper.out 2>&1 &

while true
do
  cmd=$(ps aux | grep "$JAVA_HOME/bin/java.*zookeeper-gc\.log.*" | grep -v "grep" | wc -l)
  echo $cmd
  if [ $cmd -ge 1 ]
    then
      echo Zookeeper is running
      break
    else
      echo NOT RUNNING
      sleep 1
  fi
done

# sleep a few extra seconds to make sure that the port is actually available
sleep 4
./zookeeper-shell.sh %ZOOKEEPER_CONNECT% rmr /brokers
./kafka-server-start.sh ../config/server.properties > kafka.out 2>&1 &

while true
do
    sleep 1
done

In the Automic/External.Resources/kafka directory, create a new sub-directory called zk-tmp. This directory will hold all your zookeeper data files.
Open the zookeeper.properties file located here: <automic>/External.Resources/kafka/config. Find the dataDir parameter and edit it to reflect the newly created zk-tmp directory.

Example: dataDir=/automic/External.Resources/kafka/kafka/zk-tmp
In the <automic>/External.Resources/kafka/bin directory under root user make the file executable for example: run the command chmod +x ./start.sh.

Step 3: Add the Streaming Platform to the Service Manager SMD file

Note: For steps 3 and 4 you must have Service Manager installed. If you do not have Service Manager installed, see the following links: ServiceManager, Start Parameters - ServiceManager.

If you have installed Service Manager open the uc4.smdfile found here : <automic>/Automation.Platform/ServiceManager/bin/uc4.smd. Using your defined directory path, add a DEFINE STREAMING-PLATFORM line as shown in the example below:

DEFINE STREAMING-PLATFORM;<automic>/External.Resources/kafka/kafka/bin/start.sh;

<automic>/External.Resources/kafka/kafka/bin

Step 4: Add the Streaming Platform to the Service Manager SMC file

Open the uc4.smcfile found here:<automic>/Automation.Platform/ServiceManager/bin/uc4.smc andinsert the following two lines (in bold) and close and restart the service manager or simply start the STREAMING-PLATFORM service:

WAIT 10

CREATE RULE-ENGINE

WAIT 3

CREATE ANALYTICS-DATASTORE

WAIT 14

CREATE ANALYTICS-BACKEND

WAIT 3

CREATE STREAMING-PLATFORM

Step 5: Adapt the Configuration file

Note: This step can not be completed until you install the Analytics Backend. If you have not done this, take note of the following host and port configuration.

Some adaptions must be made in the application.properties configuration file. This file can be found here: <automic>/Automation.Platform/Analytics/backend/application.properties:

Set the communication parameters for the Backend to communicate with Kafka:

kafka.bootstrap_servers=hostIP:9092

Note: The hostIP and port see the machine that Kafka is installed on.

Set the communication parameters for the Backend to communicate with Zookeeper:

zookeeper.bootstrap_servers=hostIP:2181

Notes:

The hostIP and port, check the machine that Zookeeper is installed on.
If you are using the Event Engine feature and the Streaming Platform becomes unavailable, Rules that are running on the Rule Engine will fail after a timeout. The default timeout is 5 minutes (305000 milliseconds) and it can be adapted if needed. To change the timeout, configure the following:

kafka.consumer_configs.edda_events[request.timeout.ms]=305000

Step 6: Secure the Streaming Platform (optional)

For further information see: TLS/SSL Setup between Backend and Streaming Platform