mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

Archive · DevOps

Self-Hosted Kafka 4: A Three-Node KRaft Cluster

Apache Kafka 4.0 has no ZooKeeper at all. Build a KRaft cluster with dedicated controllers and brokers, sized, tuned and tested for production, with a smaller combined-node option.

4 min read

  • Apache Kafka 4.0
  • KRaft
  • OpenJDK 17
  • Ubuntu 24.04 LTS
  • systemd

Apache Kafka 4.0.0 (released 18 March 2025) removed ZooKeeper completely, so every cluster now uses KRaft, Kafka’s own Raft-based metadata quorum. It is the newest release on 1 September 2025, with 3.9.1 (May 2025) the last release line that can migrate an existing ZooKeeper cluster to KRaft. Brokers, Connect and the command-line tools need Java 17; clients and Kafka Streams need Java 11 or newer.

Choose a topology

Topology Layout Use when
Dedicated (recommended) 3 controller nodes (metadata only) + 3 or more broker nodes Production and anything critical
Combined 3 nodes, each acting as controller and broker Small, development or low-risk environments

The Kafka documentation states that combined mode is not recommended in critical deployment environments: a broker under heavy load can slow the controller, and you cannot roll or scale the two roles separately. A quorum of 3 controllers survives one failure and 5 survive two (it needs 2N + 1 for N failures).

flowchart TB
  subgraph Z1[Zone A]
    C1[Controller 1<br/>node.id 1]
    B1[Broker 1<br/>node.id 101]
  end
  subgraph Z2[Zone B]
    C2[Controller 2<br/>node.id 2]
    B2[Broker 2<br/>node.id 102]
  end
  subgraph Z3[Zone C]
    C3[Controller 3<br/>node.id 3]
    B3[Broker 3<br/>node.id 103]
  end
  C1 <--> C2
  C2 <--> C3
  C1 <--> C3
  B1 -->|metadata| C1
  B2 -->|metadata| C2
  B3 -->|metadata| C3
  P[Producers] -->|acks all| B1
  P --> B2
  K[Consumers<br/>consumer groups] --> B3
  B1 <-->|replication| B2
  B2 <-->|replication| B3

Spread nodes across failure domains (zones, racks or hosts) and set broker.rack so replicas land in different ones.

Sizing

Disk and network drive the sizing, not CPU. Disk needed cluster-wide is roughly ingress rate × retention × replication factor, divided by a usable fraction (use about 0.7 to leave room for growth and rebalancing). For 20 MB/s of ingress, 7 days of retention and replication factor 3:

20 MB/s × 604,800 s × 3 ≈ 36 TB, and 36 TB ÷ 0.7 ≈ 52 TB, or about 17 TB per broker across three brokers.

Resource Starting point per broker
CPU and memory 8 vCPU, 32 GB RAM (a 6 GB JVM heap leaves the rest for the page cache, which Kafka relies on)
Disk Dedicated data disks on XFS or ext4, separate from the operating system
Network 10 Gbps for busy clusters; 1 Gbps is a bottleneck sooner than expected
Controllers 2 to 4 vCPU, 8 GB RAM, small fast disk for the metadata log

Prepare each node

sudo apt update && sudo apt install -y openjdk-17-jre-headless
sudo useradd -r -m -d /var/lib/kafka -s /usr/sbin/nologin kafka
sudo mkdir -p /opt /etc/kafka /var/lib/kafka/data
curl -fsSLO https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz   # verify the checksum and signature
sudo tar -xzf kafka_2.13-4.0.0.tgz -C /opt && sudo ln -s /opt/kafka_2.13-4.0.0 /opt/kafka

# kernel and limits
echo 'vm.swappiness=1' | sudo tee /etc/sysctl.d/99-kafka.conf
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.d/99-kafka.conf   # many partitions need many mappings
sudo sysctl --system
echo 'kafka - nofile 100000' | sudo tee /etc/security/limits.d/kafka.conf

Use fixed DNS names (kc1, kb1 and so on), synchronised clocks, and open port 9093 between controllers and brokers and 9092 from clients only.

Configuration

Controller (/etc/kafka/server.properties on nodes 1 to 3, changing node.id):

process.roles=controller
node.id=1
controller.quorum.voters=1@kc1:9093,2@kc2:9093,3@kc3:9093
listeners=CONTROLLER://:9093
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka/metadata

Broker (nodes 101 to 103, changing node.id, the advertised name and the rack):

process.roles=broker
node.id=101
controller.quorum.voters=1@kc1:9093,2@kc2:9093,3@kc3:9093
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kb1:9092
inter.broker.listener.name=PLAINTEXT
broker.rack=zone-a
log.dirs=/data1/kafka,/data2/kafka
num.partitions=6
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
unclean.leader.election.enable=false
log.retention.hours=168
log.segment.bytes=1073741824

The listener here is plain text so the first cluster is easy to build and verify; the next post adds TLS and SASL, and you should not expose this configuration beyond a private network.

For the combined three-node option, each node uses process.roles=broker,controller, listeners=PLAINTEXT://:9092,CONTROLLER://:9093 and the same quorum setting.

Format storage and start

Every node must be formatted once with the same cluster ID; Kafka no longer formats automatically, because auto-formatting can hide errors.

CID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)       # run once, reuse the value on every node
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format -t $CID -c /etc/kafka/server.properties

A systemd unit keeps it running and restarts it on failure:

[Unit]
Description=Apache Kafka
After=network-online.target
[Service]
User=kafka
Environment="KAFKA_HEAP_OPTS=-Xms6g -Xmx6g"
LimitNOFILE=100000
ExecStart=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target

Start the controllers first, then the brokers.

Verify before real traffic

/opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-server kb1:9092 describe --status
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kb1:9092 --create --topic smoke --partitions 6 --replication-factor 3
/opt/kafka/bin/kafka-producer-perf-test.sh --topic smoke --num-records 1000000 --record-size 1024 \
  --throughput -1 --producer-props bootstrap.servers=kb1:9092 acks=all

Then stop one broker and check that producers with acks=all keep working (with min.insync.replicas=2 and replication factor 3, one broker can fail), and that the in-sync replica set recovers after the broker returns. Do the same for one controller.

Client settings that matter

Use acks=all and idempotent producers (the default in current clients). Kafka 4.0 also makes the new consumer-group protocol (KIP-848) generally available, which reduces rebalance disruption; clients opt in with group.protocol=consumer. If you are moving from an older cluster, note that a ZooKeeper cluster cannot upgrade directly to 4.0: migrate to KRaft on 3.9 first.