KAFKA SSL WITH ZOOKEEPER Quorum

What is Kafka ?

Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which typically send the data records in simultaneously. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally.

Kafka provides three main functions to its users:

• Publish and subscribe to streams of records
• Effectively store streams of records in the order in which records were generated
• Process streams of records in real-time

Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.

Apache KAFKA Architechure:

Source:https://learnbyinsight.com/2020/07/26/beginner-guide-to-understand-kafka/

What is SSL?

SSL stands for Secure Sockets Layer and, in short, it’s the standard technology for keeping an internet connection secure and safeguarding any sensitive data that is being sent between two systems, preventing criminals from reading and modifying any information transferred, including potential personal details. The two systems can be a server and a client (for example, a shopping website and browser) or server to server (for example, an application with personally identifiable information or with payroll information)

SSL between two systems can be a server and a client

KAFKA SSL ARCHITECHURE:

SSL Communication between Zookeeper and Kafka Brokers

Setup Kafka 3 node Cluster

Install Java 8 on 3 servers and export env variables

sudo yum install java-1.8.0-openjdk-devel

Download JCE files for Java 8 - https://www.oracle.com/java/technolog...

replace in $JAVA_HOME/jre/lib/security/ on all 3 servers

Download Kafka: https://kafka.apache.org/downloads

Untar on server1,server2, and server3

Generate RootCA

openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650

copy ca-key and ca-cert to all servers

Repeat the below steps on all 3 servers. make sure to replace server-fqdn with your server fully qualified domain name

Create Truststore

keytool -keystore kafka.truststore.jks -alias server-fqdn -import -file ca-cert

Create Keystore
keytool -keystore kafka.keystore.jks -alias server-fqdn -validity 3650 -genkey -keyalg RSA -ext SAN=DNS: server-fqdn

Create certificate signing request
keytool -keystore kafka.keystore.jks -alias server-fqdn -certreq -file ca-request-zookeeper

Sign the certificate
openssl x509 -req -CA ca-cert -CAkey ca-key -in ca-request-out ca-signed -days 3650 -CAcreateserial

Import the CA into Keystore
keytool -keystore kafka.keystore.jks -alias ca-cert -import -file ca-cert

Import the signed certificate into Keystore
keytool -keystore kafka.keystore.jks -alias server-fqdn -import -file ca-signed

Modify Kafka and zookeeper properties on all 3 servers

Broker: 01

Server. properties

#SSL CONFIGURATIONS
zookeeper.connect=server-fqdn01:2182,server-fqdn02:2182,server-fqdn03:2182
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=kafka.truststore.jks
zookeeper.ssl.truststore.password=happylearning
zookeeper.ssl.keystore.location=kafka.keystore.jks
zookeeper.ssl.keystore.password=happylearning
zookeeper.set.acl=false

listeners=SSL://server-fqdn01:9093
advertised.listeners=SSL://server-fqdn01:9093
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=happylearning
ssl.keystore.location=kafka.keystore.jks
ssl.keystore.password=happylearning
ssl.key.password=happylearning #makesure key password must be same from CA ROOT KEY
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.client.auth=required
security.inter.broker.protocol=SSL
ssl.endpoint.identification.algorithm=

same on server 2 and server 3. Just change listeners SSL fqdn on servers 2 and 3

zookeeper. properties

admin.enableServer=true
admin.serverPort=9090
tickTime=2000
initLimit=5
syncLimit=2

#clientPort=2181
secureClientPort=2182
maxClientCnxns=0
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.trustStore.location=/kafka.truststore.jks
ssl.trustStore.password=happylearning
ssl.keyStore.location=kafka.keystore.jks
ssl.keyStore.password=happylearning
ssl.clientAuth=need

server.1=server-fqdn01:2888:3888
server.2=server-fqdn02:2888:3888
server.3=server-fqdn03:2888:3888

producer. properties

bootstrap.servers=server-fqdn01:9093,server-fqdn02:9093,server-fqdn03:9093
security.protocol=SSL
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=happylearning
ssl.keystore.location=kafka.keystore.jks
ssl.keystore.password=happylearning
ssl.key.password=happylearning

consumer. properties

bootstrap.servers=server-fqdn01:9092
group.id=ssl-consumer-group
security.protocol=SSL
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=happylearning
ssl.keystore.location=kafka.keystore.jks
ssl.keystore.password=happylearning
ssl.key.password=happylearning

Let's start below commands on 3 servers based on broker ID in the server.properties files sequence oirder

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

Start Zookeeper:

cd $KAFKA_HOME

bin/zookeeper-server-start.sh -daemon ../config/zookeeper.properties

Start Kafka service:

nohup ./kafka-server-start.sh ../config/server.properties &

Create Topic

./kafka-topics.sh — bootstrap-server server-fqdn01:9093 — create — replication-factor 1 — partitions 2 — command-config ../config/consumer.properties — topic test-narendra

Lets Test Producer and Consumer:

Producer:

./kafka-console-producer.sh — broker-list server-fqdn01:9093,server-fqdn01:9093,server-fqdn01:9093 — topic test-narendra — producer.config ../config/producer.properties

HELLO THIS IS NARENDRA SSL ENABLED KAFKA

Consumer:

/kafka-console-consumer.sh — bootstrap-server server-fqdn01:9093 — topic test-tilak — from-beginning — consumer.config ../config/consumer.properties
HELLO THIS IS NARENDRA SSL ENABLED KAFKA

References:

Configure Confluent Platform to use TLS/SSL encryption and authentication | Confluent Documentation

Preparing SSL certificates for Apache Kafka

https://kafka.apache.org/20/documentation/streams/developer-guide/security.html