A cluster on a private network still needs authentication, encryption and monitoring. This post hardens the KRaft cluster from the previous post and covers routine operations. The settings apply to Kafka 4.0 and to Kafka 4.1, which was released on 2 September 2025.
Security design
flowchart TB
P[Producers and consumers<br/>SCRAM user with TLS] -->|SASL SSL 9092| B[Brokers]
A[Admin tools<br/>admin user] -->|SASL SSL 9092| B
B <-->|mutual TLS 9093| C[Controllers]
B <-->|SASL SSL replication| B2[Other brokers]
CA[Private CA] -.->|issues certificates| B
CA -.-> C
B --> ACL[ACL authorizer<br/>StandardAuthorizer]
PR[Prometheus] -->|JMX exporter 7071| B
PR --> C
| Channel | Protection |
|---|---|
| Clients to brokers | TLS encryption and SASL/SCRAM-SHA-512 authentication |
| Broker to broker | The same SASL_SSL listener, with an admin identity |
| Broker and controller | Mutual TLS on the controller listener |
| Authorisation | ACLs per user, deny by default |
Certificates
Issue one certificate per node with its DNS names as subject alternative names, and keep the CA key offline:
openssl req -new -newkey rsa:4096 -nodes -keyout kb1.key -out kb1.csr \
-subj "/CN=kb1.corp.example" -addext "subjectAltName=DNS:kb1.corp.example,DNS:kb1"
openssl x509 -req -in kb1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 \
-copy_extensions copy -out kb1.crt
openssl pkcs12 -export -in kb1.crt -inkey kb1.key -certfile ca.crt -name kb1 -out kb1.p12
keytool -importcert -noprompt -alias ca -file ca.crt -keystore ca.p12 -storetype PKCS12Track expiry dates and rotate before they pass; Kafka can reload broker keystores dynamically through kafka-configs.sh, which avoids a restart.
Broker configuration
listeners=SASL_SSL://:9092,CONTROLLER://:9093
advertised.listeners=SASL_SSL://kb1.corp.example:9092
listener.security.protocol.map=SASL_SSL:SASL_SSL,CONTROLLER:SSL
inter.broker.listener.name=SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="<from a secret store>";
ssl.keystore.type=PKCS12
ssl.keystore.location=/etc/kafka/tls/kb1.p12
ssl.keystore.password=<from a secret store>
ssl.truststore.type=PKCS12
ssl.truststore.location=/etc/kafka/tls/ca.p12
ssl.truststore.password=<from a secret store>
listener.name.controller.ssl.client.auth=required
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin;User:CN=kb1.corp.example;User:CN=kb2.corp.example;User:CN=kb3.corp.examplesuper.users must include the admin account and the identities of the broker certificates, because the controller listener authenticates brokers by the subject of their certificate. Apply the controller listener settings on the controller nodes in the same way.
The first SCRAM user in KRaft. Brokers need a SCRAM identity to talk to each other before any user can be created through the cluster, so add the admin credential when you format storage (Kafka’s security documentation describes this option):
bin/kafka-storage.sh format -t $CID -c /etc/kafka/server.properties \
--add-scram 'SCRAM-SHA-512=[name=admin,password=<secret>]'Create further users through the running cluster, using an admin client configuration file (admin.properties) with the TLS and SASL settings:
bin/kafka-configs.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties \
--alter --add-config 'SCRAM-SHA-512=[password=<secret>]' --entity-type users --entity-name orders-svc
bin/kafka-acls.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties \
--add --allow-principal User:orders-svc --operation Read --operation Write --operation Describe --topic orders
bin/kafka-acls.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties \
--add --allow-principal User:orders-svc --operation Read --group orders-consumersAdd quotas so one noisy client cannot starve others:
bin/kafka-configs.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties \
--alter --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \
--entity-type users --entity-name orders-svcMonitoring
Attach the Prometheus JMX exporter to each broker and controller as a Java agent:
KAFKA_OPTS="-javaagent:/opt/jmx_prometheus_javaagent.jar=7071:/etc/kafka/jmx-rules.yml"| Signal (JMX name) | Alert when |
|---|---|
UnderReplicatedPartitions |
Above zero for 5 minutes |
OfflinePartitionsCount |
Above zero (page immediately) |
ActiveControllerCount |
Total across the cluster is not exactly 1 |
Consumer group lag (kafka-consumer-groups.sh --describe, or an exporter such as Kafka Exporter) |
Growing for 15 minutes |
RequestHandlerAvgIdlePercent |
Below 30 percent |
| Disk used on data volumes | Above 75 percent |
KRaft metrics (raft-metrics: leader, commit latency) |
Leader changes frequently, or commit latency climbs |
| Certificate expiry | Within 21 days |
Prometheus metric names depend on the rules in your exporter file, so write the alert expressions against the names it actually produces. Keep dashboards for throughput per topic, request latency, and ISR shrink and expand rates.
Day-two operations
Rolling restarts and upgrades. Change one node at a time. In KRaft, upgrade the controllers first, then the brokers, and after every node runs the new version, raise the metadata version:
bin/kafka-features.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties upgrade --metadata 4.1Between nodes, wait until under-replicated partitions return to zero. Read the release notes first; the 4.0 line removed older protocol versions and MirrorMaker 1, and a 3.x ZooKeeper cluster must migrate to KRaft on 3.9 before moving to 4.x.
Adding brokers and rebalancing. New brokers get no partitions until you move some:
bin/kafka-reassign-partitions.sh --bootstrap-server kb1.corp.example:9092 --command-config admin.properties \
--generate --topics-to-move-json-file topics.json --broker-list 101,102,103,104
bin/kafka-reassign-partitions.sh ... --execute --reassignment-json-file plan.json --throttle 50000000
bin/kafka-reassign-partitions.sh ... --verify --reassignment-json-file plan.jsonAlways set a throttle, run during quiet hours, and remove the throttle after --verify reports completion.
Retention. Set retention by bytes (retention.bytes) as well as time (retention.ms), so a surge in traffic cannot fill the disks, and alert before volumes reach 75 percent.
Disaster recovery. Replication inside one cluster protects against node loss, not against losing a site or an operator mistake. For a second site, run MirrorMaker 2 (it replicates topics, consumer group offsets and ACL-relevant metadata between clusters). Decide the recovery point and recovery time you need, practise the failover, and back up configuration, certificates and the ACL list outside the cluster.
Runbook starting points
| Symptom | First checks |
|---|---|
| Producers time out | Broker network and disk saturation; min.insync.replicas not met because a broker is down |
| Consumer lag rising | Slow consumers, rebalances, a hot partition; compare produce rate and consume rate |
| Under-replicated partitions | A slow or dead broker, a full disk, a throttled reassignment |
| Authentication failures | Expired certificate, wrong SCRAM user, missing ACL (check broker authorizer logs) |
| Controller election loops | Network partitions between controllers, clock drift, disk latency on the metadata log |