TLS/SSL Setup between Backend and Streaming Platform
Introduction
The following example that is outlined describes how to configure TLS/SSL between the Analytics Backend and the Streaming Platform (Apache Kafka).
Note: It is advised to use a secure configuration if you have installed Analytics Backend and the Streaming Platform on a separate network.
This page includes the following:
Prerequisites
A generated Java Keystore is required for each deployed node in your Streaming Platform (Kafka) cluster. See: https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores
More Resources
The official Apache Kafka website provides the following information to help setup and configure SSL. It should be used to supplement the example describe in this document. See: http://kafka.apache.org/documentation.html#security_ssl_key
Note: The following details are based on a Streaming Platform (Kafka) single broker node.
Getting Started
The first step is to configure the Streaming Platform to allow clients (that is, Backend) to connect over SSL.
The following bash script sums-up the first three steps that are described on the Apache Kafka web page.
- Generate TLS/SSL key and certificate for each Streaming Platform (Kafka) broker
- A certificate is generated using the Java keytool.
- Two important parts to this step are the Keystore (stores the certificate) and Validity (number of days the certificate is valid e.g. "365")
- Creating a CA (Certificate Authority)
- This is a public-private key pair and certificate to sign other certificates.
- Signing the certificate
- Sign generated certificate for Streaming Platform (Kafka) broker with generated CA.
Script example:
#!/bin/bash
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:automic
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
Generating a TLS/SSL key and a self-signed certificate, create a CA and sign a certificate
The commands in the example script above (Steps 1,2 and 3) were separately launched (in a working directory) on the single Streaming Platform (Kafka) broker. Here is the output:
Step 1 -Generate SSL key
[root@vmsdlbroker01 ssl_vmsdlbroker01]# keytool -keystore server.keystore.jks -alias vmsdlbroker01 -validity 365 -keyalg RSA -genkey
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: vmsdlbroker01
What is the name of your organizational unit?
[Unknown]:
What is the name of your organization?
[Unknown]:
What is the name of your City or Locality?
[Unknown]:
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]:
Is CN=vmsdlbroker01, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?[no]: yes
Enter key password for <vmsdlbroker01>
(RETURN if same as keystore password):
Step 2 - Create a CA
[root@vmsdlbroker01 ssl_vmsdlbroker01]# openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
Generating a 2048 bit RSA private key
.......................................................+++
writing new private key to 'ca-key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:vmsdlbroker01
Email Address []:
Step 3 - Sign the certificate
[root@vmsdlbroker01 ssl_vmsdlbroker01]# keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
Enter keystore password:
Re-enter new password:
Owner: CN=vmsdlbroker01, O=Default Company Ltd, L=Default City, C=XX
Issuer: CN=vmsdlbroker01, O=Default Company Ltd, L=Default City, C=XX
Serial number: e5598a57a2148d89
Valid from: Fri Aug 04 11:41:22 CEST 2017 until: Sat Aug 04 11:41:22 CEST 2018
Certificate fingerprints:
MD5: 4C:3C:66:53:FC:7A:5F:34:F5:69:03:53:5B:AE:CD:51
SHA1: F6:5A:37:7A:2E:6E:7E:2F:B9:75:7E:06:A6:A4:5F:1F:0E:38:EF:CB
SHA256:8E:BF:85:DF:ED:E8:B0:18:C7:DE:9A:21:68:99:A5:E5:AE:96:0D:C8
:D3:D8:C0:68:69:BD:5F:71:6A:A3:F9:10
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 77 BC 1A 96 24 57 EF 2B 54 51 15 76 09 51 28 D3 w...$W.+TQ.v.Q(.0010: FA 7E 6E 1D..n.]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [KeyIdentifier [0000: 77 BC 1A 96 24 57 EF 2B 54 51 15 76 09 51 28 D3 w...$W.+TQ.v.Q(.
0010: FA 7E 6E 1D ..n]]
Trust this certificate? [no]: yes
Certificate was added to keystore
[root@vmsdlbroker01 ssl_vmsdlbroker01]# keytool -keystore server.keystore.jks -alias vmsdlbroker01 -import -file cert-signed
Enter keystore password:
Certificate reply was installed in keystore
[root@vmsdlbroker01 ssl_vmsdlbroker01]#
[root@vmsdlbroker01 ssl_vmsdlbroker01]# ls -l
total 32
-rw-r--r--. 1 root root 1285 Aug 4 11:41 ca-cert
-rw-r--r--. 1 root root 17 Aug 4 11:43 ca-cert.srl
-rw-r--r--. 1 root root 1834 Aug 4 11:41 ca-key
-rw-r--r--. 1 root root 1105 Aug 4 11:43 cert-file
-rw-r--r--. 1 root root 1200 Aug 4 11:43 cert-signed
-rw-r--r--. 1 root root 970 Aug 4 11:42 client.truststore.jks
-rw-r--r--. 1 root root 4059 Aug 4 11:44 server.keystore.jks
-rw-r--r--. 1 root root 970 Aug 4 11:41 server.truststore.jks
Configuring the Streaming Platform (Kafka) Broker
To configure the Streaming Platform (Kafka) broker the server.properties file should be edited. The file is located in the following directory:
<Automic>/External.Resources/kafka/kafka/config
Note: The server.keystore.jks file is the previously generated self-signed certificate for this broker.
listeners=PLAINTEXT://vmsdlbroker01:9092,SSL://vmsdlbroker01:9093
########################
## SSL Configuration ##
########################
ssl.keystore.location=/opt/kafka/kafka/ssl vmsdlbroker01/server.keystore.jks
ssl.keystore.password=automic
ssl.key.password=automic
Configuring the Streaming Platform (Kafka) Client (IA Agent)
To configure the Streaming Platform (Kafka) client (the Backend) the application.properties file should be edited. The file is located in the following directory:
<automic>/Automation.Platform/Analytics/backend
Note: The keystore.jks file should be generated on the Backend side using the same set of tools.
########################
## SSL Configuration ##
########################
# By enabling the server.ssl.* settings the Backend will only accept HTTPS connections. An unsecure
# HTTP connection will be automatically disabled. Change the port to 8443 if required by setting
server.port=8443
# Path to the key store that holds the TLS/SSL certificate (typically a .jks file).
server.ssl.key-store=/automic/Automation.Platform/Analytics/backend/keystore.jks
# Password used to access the key store.
server.ssl.key-store-password=automic
# Password used to access the key in the key store.
server.ssl.key-password=automic
## Kafka ##
# Specify Kafka hosts
kafka.bootstrap_servers=vmsdlbroker01:9093
kafka.producer_configs.default[security.protocol] = SSL
kafka.consumer_configs.default[security.protocol] = SSL
kafka.producer_configs.default[ssl.truststore.location] = /automic/SSL/kafka/client.truststore.jks
kafka.consumer_configs.default[ssl.truststore.location] = /automic/SSL/kafka/client.truststore.jks
kafka.producer_configs.default[ssl.truststore.password] = automic
kafka.consumer_configs.default[ssl.truststore.password] = automic
# Specify Zookeeper hosts
zookeeper.bootstrap_servers=vmsdlbroker01:2181
The client.trustore.jks is created on the Streaming Platform (Kafka) broker side (vmsdlbroker01) and put into the trusted local directory (/automic/SSL) on the client side (Backend).
Note: The Streaming Platform broker and client should now be able to communicate securely.