Setting Up the ESP Connector

Integrating AAI and ESP Workload Automation allows you to source event data and job definitions from ESP and make it available in AAI to optimize workload tracking and obtain improvement analytics.

As an AAI administrator, you set up the ESP Connector. You also add an ESP scheduler that connects to the relevant ESP environment. The scheduler is a representation of the ESP system that provides the data. AAI uses that data to build jobstreams, perform calculations, predict outcomes, populate charts, generate reports and so forth.

The AAI / ESP integration is unique. It requires some work in USS and an AAI connector to be installed in a distributed system, similar to the Automic Automation and Control-M integrations. However, it also requires the deployment of a number of components on the mainframe, similar to the CA7 and IWSz integrations.

This section guides you through the necessary steps to install and configure it.

Who Should Read These Topics?

Integrating ESP and AAI requires the collaboration between technical teams that support ESP and AAI administrators. To accomplish these tasks, a deep understanding of distributed systems, mainframe technology and AAI is required.

Overview

The ESP Connector resides in the distributed system and establishes the communication between ESP and AAI, it submits data request to ESP and handles deliveries in return. It is a stand-alone component and, as such, it runs in its own process space, has its own installer and writes its own log files.

Before integrating an ESP instance with your AAI system, you must have AAI up and running. For more information, see Installation and Upgrade.

If you need help with the installation, please contact our support team at https://support.broadcom.com/.

The naming conventions for the ESP Connector .zip file are AAI.ESP_Connector_<product_version>.zip.

Also, you can secure your Connectors using TLS 1.2 based secure connections. For more information, see Securing AAI

See also Configuring the Connectors for TLS Authentication.

Integration Requirements

Before installing the ESP Connector, make sure that all the necessary ESP Workload Automation components are installed and configured. For more information, please refer to the official ESP Workload Automation documentation.

As an AAI administrator, you need to understand the role of the ESP Connector in the integration; therefore, you need to be aware of the different components involved in the process:

• Relevant PTFs, which contain the ESP Adapter and the changes to the ESP Master subsystem

• LiveData Service in ESP, an internal component which feeds operational data to AAI in real-time

• ESP Adapter, which gets data from the mainframe, creates definitional and operational data and passes it on to the Message Service Hub

• Message Service Hub, which coordinates the movement of (high volume) data

• ESP Connector, submits request for definition data to ESP in MQ format and handles deliveries

ESP Connector Requirements

Before starting with the ESP Connector installation, make sure that your system meets the following requirements:

• Minimum memory amount: 4GB

• Minimum disk space: 1GB

• Java is installed and PATH contains a path to Java binaries, otherwise configuring the ESP Connector is not possible because the configuration script fails.

Installing/Upgrading the ESP Connector on Windows

The configuration of the ESP connector starts automatically after the installation.

To install or upgrade the ESP Connector on Windows:

1. Go to https://downloads.automic.com/, log in with your Broadcom account and download the relevant version of the Connector. For more information ,see Downloads.

2. Put the .zip file in a temp location on the server where the connector will run and unzip the file to that location.

   The Connector is delivered as a mountable ISO file within the .zip file that you downloaded.

3. Mount the ISO file. To do so, double-click the file, which is then mounted automatically as the new CD-ROM drive by default.

4. Once the ISO file is mounted, run the esp-aai-connector-<version>.exe file within to setup the installer.

5. Follow the instructions in the setup wizard. You are prompted to configure the connector during the installation.

   Make sure you enter the information required:

   • AAI server URI

     You must specify the protocol, domain host name or IP address, and a listening port number of the AAI server. This information should be captured during the AAI server installation.

     If you are using the Connector certificate, this URI may use HTTPS and port 4443. Otherwise, it will likely be the same port that is used to launch the AAI client from a browser, for example http{s}://hostname:port.

   • Connector name

     Name of your choice. This name will be selected from the web UI when adding a scheduler to AAI.

   • Host name

     Domain name or IP address of the host where the Connector runs

   • Connector key password

     This is only required if an HTTPS connection is used between the Connector and the AAI server. If HTTP is used, leave this empty and hit Enter.

   • AAI server user name

     Only required when using HTTP.

     Name of the user ID that will be used to establish a connection to the AAI server. The user must be authorized to add new schedulers.

   • AAI server password

     This is required only if a HTTP connection is used between the Connector and the AAI server. If HTTPS is used, this can be omitted.

   • Connector domain

     Typically this is JAWS. This is the login domain for the user/password provided and it can be any of the list of domains that have been set up in AAI. For more information about domains, see Domains.

   • Log file output path

     The path to the location of the log files. The directory must exist and be writable for the connector user account.

   • Max log file size

     The maximum size of the log files. Must be defined as an integer in MB and must be 1 or larger.

   • Max log file count

     The number of log files to be retained. Must be defined as an integer in MB and must be 5 or larger.

   Once the installation completes successfully, the ESP connector starts as a Service.

   If the connection is successful, the application.yml file is created in the folder where you have installed the Connector. This is a configuration file that contains the configuration values that you have just entered. At the same time, a backup copy of the application.yml is automatically created and renamed to application.yml.bak.

   In the case of a failure, the application.yml file is not populated with the values you entered. You would have to manually repeat the configuration process.

6. If the connector configuration failed during the installation, or if you want to change the configuration, do the following:

   1. Go to the folder where the connector is installed.

   2. Execute setup.bat file. This will open the Command prompt window.

      Important!

      • Once setup.bat is started, the following message prompts you to obtain the Connector certificate: "If you are planning to make a TLS based connection to AAI then before proceeding please obtain the connector certificate package from the AAI server and copy it to C:\Program Files\esp-aai-connector\secure\"

      • If using the Connector certificate, generate it now. For more information, see Generating the AAI Server Key.

   3. Follow the instructions and enter the required information, as mentioned above.

   4. After you have entered the required information, the script continues with the connection to the AAI server to verify that the provided data is correct.

You also have to add ESP schedulers to your AAI environment. To do so, go to System > Schedulers in your web interface and follow the steps in the dialog to add a ESP scheduler. For more information, see Configuring ESP Workload Automation Schedulers.

If the process was successful, the new schedulers are listed with all the schedulers available in your system.

You must add a new ESP scheduler for each ESP instance that you want to integrate into your AAI environment. You can add the ESP schedulers in AAI before installing the ESP Connector. However, you will not be able to use them until the Connector is up and running.

Starting, Stopping, Uninstalling and Connector Status

On Windows, you can start and stop the ESP connector and check its status as a Service.

You can also run the esp-aai-connector.exe to start it, stop it and check its status.

esp-aai-connector.exe /start

esp-aai-connector.exe /stop

esp-aai-connector.exe /status

Expected output if the service is running:

Running.

Auto-start.

Expected status if the service is not running:

Not running.

Auto-start.

For details, refer to the output.log file that is created in the folder where the Connector was installed.

Uninstalling the Connector

To uninstall the connector, do the following:

1. Go to Apps & features.

2. Select esp-aai-connector-(version).exe.

3. Select the Uninstall button and follow the instructions.

Installing the ESP Connector on UNIX (Linux or zLinux)

To install the ESP Connector on UNIX (Linux or zLinux):

1. Go to https://downloads.automic.com/, log in with your Broadcom account and download the relevant version of the Connector. For more information ,see Downloads.

2. Unzip the .zip file that you downloaded.

   The Connector is delivered as a mountable ISO file within the .zip file that you downloaded.

3. Mount the ISO file. To do so, create a mount point using the sudo mkdir -p /mnt/iso command. Then mount the ISO file using the mount -o loop <path_to_iso> /mnt/iso mount command .

   To verify if the file has been mounted correctly use the ls -al /mnt/iso command.

   Example

   [root@hostname ~]$ sudo mkdir -p /mnt/iso

   [root@hostname ~]$ sudo mount -o loop /home/root/esp-connector-1.0.2.iso /mnt/iso

   mount: /dev/loop0 is write-protected, mounting read-only

   [root@hostname ~]$ ls -al /mnt/iso/

   total 179663

   drwxr-xr-x. 2 root root 2048 Feb 8 11:35 .

   drwxr-xr-x. 3 root root 17 Mar 10 13:45 ..

   -rwxr-xr-x. 1 root root 48238080 Feb 8 11:35 esp-aai-connector-1.0.2.exe

   -rw-r--r--. 1 root root 45244443 Feb 8 11:35 esp-aai-connector-1.0.2.noarch.tar.gz

   -rw-r--r--. 1 root root 45244455 Feb 8 11:35 esp-aai-connector-1.0.2.s390x.tar.gz

   -rw-r--r--. 1 root root 45244451 Feb 8 11:35 esp-aai-connector-1.0.2.x86_64.tar.gz

   [root@hostname ~]$

4. After mounting the ISO file, you can unpack the .tar.gz file within to get the connector-install.sh and esp-aai-connector~<version>.<architecture>.rpm files.

5. Run connector-install.sh to start the installation, for example:

   ./connector-install.sh install

   The Connector is installed using the .rpm file provided.

6. Configure the Connector to point to your AAI installation. To do so, run the configure script:

   cd /opt/broadcom/esp-aai-connector

   sudo -u esp-aai-connector ./configure-esp-aai-connector.sh

   Follow the prompts and make sure you enter the information required:

   • AAI server URI

     You must specify the protocol, domain host name or IP address, and a listening port number of the AAI server. This information should be captured during the AAI server installation.

     If you are using the Connector certificate, this URI may use HTTPS and port 4443. Otherwise, it will likely be the same port that is used to launch the AAI client from a browser, for example http{s}://hostname:port.

   • Connector name

     Name of your choice. This name will be selected from the web UI when adding a scheduler to AAI.

   • Host name

     Domain name or IP address of the host where the Connector runs

   • Connector key password

     This is only required if an HTTPS connection is used between the Connector and the AAI server. If HTTP is used, leave this empty and hit Enter.

   • AAI server user name

     Only required when using HTTP.

     Name of the user ID that will be used to establish a connection to the AAI server. The user must be authorized to add new schedulers.

   • AAI server password

     This is required only if a HTTP connection is used between the Connector and the AAI server. If HTTPS is used, this can be omitted.

   • Connector domain

     Typically this is JAWS. This is the login domain for the user/password provided and it can be any of the list of domains that have been set up in AAI. For more information about domains, see Domains.

   • Log file output path

     The path to the location of the log files. The directory must exist and be writable for the connector user account.

   • Max log file size

     The maximum size of the log files. Must be defined as an integer in MB and must be 1 or larger.

   • Max log file count

     The number of log files to be retained. Must be defined as an integer in MB and must be 5 or larger.

   Example

   This example shows the Connector installed running HTTPS and on the default port. The Connector name can be freely defined and shows up in the UI when adding the scheduler.

   [sudo -u esp-aai-connector ./configure-esp-aai-connector.sh
   If you are planning to make a TLS based connection to AAI then before proceeding
   please obtain the connector certificate package from the AAI server
   and copy it to ./secure/
   Press ENTER to continue...
   
   *** CONFIGURING CONNECTOR ***
   
   Backing up configuration file to: application.yml.bak...
   ...Configuration file backed up to: application.yml.bak.
   
   Please enter the following configuration parameters.
   
   AAI server URI [https://localhost:4443]: https://<host>:4443
   Connector name: esp-aai
   Host name: host
   Connector key password (will not echo, required for HTTPS):
   AAI server username (required for HTTP):
   AAI server password (will not echo, required for HTTP):
   Connector domain: JAWS
   Log file output path. The directory must exist and be writable for the connector user account. [./logs]:
   Max log file size. An integer, in MB. Must be >= 1. [10]:
   Max log file count. An integer, in MB. Must be >= 5. [5]:

   If the installation ended successfully, you see the following:

   Verifying AAI server connectivity (please stand by)...2020-12-08 23:54:55,708 INFO [ApiCallerUtil] Attempting HTTP connection to AAI
   
   ...AAI server connectivity verified.
   
   Saving connector configuration to: application.yml...
   ...connector configuration saved to: application.yml.
   
   *** CONNECTOR CONFIGURED ***

7. Start the service using the following command:

   sudo systemctl start esp-aai-connector

You also have to add ESP schedulers to your AAI environment. To do so, go to System > Schedulers in your web interface and follow the steps in the dialog to add a ESP scheduler. For more information, see Configuring ESP Workload Automation Schedulers.

If the process was successful, the new schedulers are listed with all the schedulers available in your system.

You must add a new ESP scheduler for each ESP instance that you want to integrate into your AAI environment. You can add the ESP schedulers in AAI before installing the ESP Connector. However, you will not be able to use them until the Connector is up and running.

Starting, Stopping, Upgrading, Uninstalling and Connector Status

You can verify the service by running the following command:

sudo systemctl status esp-aai-connector

You can also use the systemclt command to stop or to restart the connector, or to get its status. For example:

sudo systemctl stop esp-aai-connector

sudo systemctl restart esp-aai-connector

sudo systemctl status esp-aai-connector

If for any reason the service is disabled, you can use the following systemctl command to enable it:

sudo systemctl enable esp-aai-connector

This is recommended to have the Connector restart automatically as part of the server reboot process.

You can also run the connector-install.sh to upgrade or uninstall the connector, or to get its status using the following commands:

./connector-install.sh upgrade

./connector-install.sh uninstall

./connector-install.sh status

If the current status of your system does not allow you to run a specific command, you get a corresponding error message. For example, you cannot upgrade a package that has not yet been installed.

./connector-install.sh upgrade

Can't upgrade the connector 'esp-aai-connector" isn't installed. Please 'install' instead.

Setting the Connector to Request Definition Data at Start

Optionally, after installing and configuring the connector, you can verify that the JCL is submitted to ESP before starting the ESP connector:

1. Obtain the ESP Connector ID.

   When the Connector starts, it prints out its ID in the log, for example *** Connector received (AXlmKbmkOnQ=).

   The string in brackets is the Connector ID. Keep this ID at hand because you will need it later.

2. Retrieve the scheduler configuration using the following command:

   curl http://{host:port}/aai/api/v2/schedulers?fetch=all

   where you replace host and port with your installation data.

   The output contains the configuration data of all your schedulers.

3. Use the Connector ID that you have just obtained to find the configuration that pertains to your Connector.

   The configuration data also provides the schedulerID. Keep this ID at hand because you will need it later.

4. In your schedule's configuration, copy the entire json block that is called schedulerConnectionConfig and paste it to the conf.json file. Wrap that block in { }. This is what it should look like:

   {
     "schedulerConnectionConfig" : {
       ...    
     }  
   }

5. To add the configuration that enables to retrieve jobs at start, copy and paste the following template at the end of the schedulerConnectionConfig block in your conf.json file:

   "default-trigger.publish-jobs-on-startup": true

   The content of the conf.json file should be similar to this now:

   {
     "schedulerConnectionConfig" : {
       ...
       "default-trigger.publish-jobs-on-startup": true
     }  
   }

6. To update the configuration, run the following command:

   curl -v -H "Content-Type: application/json" -X PUT --data

   @conf.json -u your_username:your_password

   http://{host:port}/aai/api/v2/schedulers/{schedulerId}

   Where:

   • {schedulerId} is the scheduler ID that you have just got in the configuration data.

   • your_username and your_password are your AAI credentials

     Note:

     If you are running this command from a different folder then specify an absolute path to your conf.json file.

   • {host:port} are your installation data

7. Once you have verified the connection, you can turn off the retrieval of jobs at start by modifying the conf.json and turning off the publish-jobs-on-startup flag. For this purpose, run again the following command:

   curl -v -H "Content-Type: application/json" -X PUT --data

   @conf.json -u your_username:your_password

   http://{host:port}/aai/api/v2/schedulers/{schedulerId}

   The content of the conf.json file should be similar to this now:

   {
     "schedulerConnectionConfig" : {
       ...
       "default-trigger.publish-jobs-on-startup": false
     }  
   }

Enabling Secure Connection Between the ESP Connector and Message Service Hub

1. Obtain the ESP Connector ID.

   When the Connector starts, it prints out its ID in the log, for example *** Connector received (AXlmKbmkOnQ=).

   The string in brackets is the Connector ID. Keep this ID at hand because you will need it later.

2. Retrieve the scheduler configuration using the following command:

   curl http://{host:port}/aai/api/v2/schedulers?fetch=all

   where you replace host and port with your installation data.

   The output contains the configuration data of all your schedulers.

3. To configure a secure connection, copy and paste the following template at the end of the schedulerConnectionConfig block in your conf.json file:

   "esp-scheduler.mq-transport": "jks",

   "esp-scheduler.mq-jks-keyStore": "",

   "esp-scheduler.mq-jks-keyStorePassword": "",

   "esp-scheduler.mq-jks-trustStore": "",

   "esp-scheduler.mq-jks-trustStorePassword": ""

4. The content of the conf.json file should be similar to this now:

   {
     "schedulerConnectionConfig": {
       ...
       "esp-scheduler.mq-transport": "jks",
       "esp-scheduler.mq-jks-keyStore": "",
       "esp-scheduler.mq-jks-keyStorePassword": ""
       "esp-scheduler.mq-jks-trustStore": "",
       "esp-scheduler.mq-jks-trustStorePassword": ""
     }  
   }

5. Update the keyStore, the trustStore and their respective passwords.

   Update the properties according to your environment. Typically, the keyStore stores the private key of the ESP Connector and the trustStore stores the public key of the Message Hub server.

   The content of the config.json file should be similar to this:

   {
     "schedulerConnectionConfig": {
       ...
       "esp-scheduler.mq-transport": "jks",
       "esp-scheduler.mq-jks-keyStore": "/u/a/keystore.jks",
       "esp-scheduler.mq-jks-keyStorePassword": "123"
       "esp-scheduler.mq-jks-trustStore": 
   "/u/a/truststore.jks",
       "esp-scheduler.mq-jks-trustStorePassword": ""
     }  
   }

6. To update the configuration, run the following command:

   curl -v -H "Content-Type: application/json" -X PUT --data

   @conf.json -u your_username:your_password

   http://{host:port}/aai/api/v2/schedulers/{schedulerId}

7. Start the ESP connector.

See also:

• Connectors for Universal Schedulers
• Configuring the Connectors for TLS Authentication
• Configuring ESP Schedulers