Apache Kafka 3.7: Installing a KRaft Broker Without ZooKeeper

Apache Kafka is a distributed log used for event streaming. Older guides start ZooKeeper first. Modern Kafka uses KRaft, where the brokers manage their own metadata with a Raft quorum, so there is nothing else to run.

Versions in use on this date. Kafka 3.7.0 was released on 27 February 2024. KRaft has been production-ready for new clusters since 3.3, and migration of existing ZooKeeper clusters became production-ready in 3.6. ZooKeeper mode is deprecated and is planned for removal in Kafka 4.0. Kafka 3.7 also publishes an official Docker image (apache/kafka). Check the Apache downloads page for any 3.7.x patch release before installing.

Target architecture

flowchart TB
  subgraph DEV[Learning setup - one node]
    N1[Kafka node<br/>roles broker and controller]
  end
  subgraph PROD[Production shape]
    C1[Controller 1] --- C2[Controller 2]
    C2 --- C3[Controller 3]
    B1[Broker 1] --> C1
    B2[Broker 2] --> C2
    B3[Broker 3] --> C3
  end
  P[Producers] -->|9092 or TLS 9093| B1
  CG[Consumer groups] --> B2
  B1 --- B2
  B2 --- B3
  B1 --> D1[(Dedicated log disk)]

A single node with process.roles=broker,controller is fine for learning. In production run three controllers (an odd number, so the quorum survives one failure) and at least three brokers, on separate hosts or availability zones. Small clusters sometimes keep controller and broker roles combined on three nodes.

Install

sudo apt update && sudo apt install -y openjdk-17-jre-headless
sudo useradd -r -m -d /opt/kafka -s /usr/sbin/nologin kafka
cd /tmp
curl -LO https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz
curl -LO https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz.sha512
sha512sum kafka_2.13-3.7.0.tgz     # compare with the published checksum
sudo tar -xzf kafka_2.13-3.7.0.tgz -C /opt/kafka --strip-components=1
sudo mkdir -p /var/lib/kafka/data && sudo chown -R kafka:kafka /opt/kafka /var/lib/kafka

Configure

The sample config/kraft/server.properties writes data to /tmp/kraft-combined-logs, which is deleted on reboot. Always change log.dirs. Create /opt/kafka/config/kraft/server.properties for a single node:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://kafka1.example.com:9092
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
log.dirs=/var/lib/kafka/data
num.partitions=3
default.replication.factor=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168

advertised.listeners is the address clients are told to reconnect to; a wrong value is the most common reason that a client connects, then hangs. For a three-node cluster, set a unique node.id on each, list all voters (1@k1:9093,2@k2:9093,3@k3:9093) and raise replication factors to 3 with min.insync.replicas=2.

Format storage once with a cluster ID (the same ID on every node):

CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format -t $CLUSTER_ID -c /opt/kafka/config/kraft/server.properties

systemd service

# /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka (KRaft)
After=network.target

[Service]
User=kafka
Environment="KAFKA_HEAP_OPTS=-Xms1g -Xmx1g"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=100000

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now kafka
sudo journalctl -u kafka -f

Keep the heap modest (Kafka relies on the OS page cache, so leave most RAM free) and put log.dirs on a dedicated XFS or ext4 disk.

First topic and a smoke test

cd /opt/kafka
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
echo "order-1" | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning --max-messages 1

Security before production

PLAINTEXT is for labs. Use TLS (SSL or SASL_SSL listeners) with SASL/SCRAM or mTLS for authentication, and enable ACLs with authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer (the KRaft authorizer). Firewall 9092 and 9093 to known networks.

Failure modes

Fails with “Invalid cluster.id”: the data directory was formatted with a different cluster ID; reformat only if the data is disposable.
Quorum never forms: voters list differs between nodes, or port 9093 is blocked.
Disk fills: set retention by time and size (log.retention.bytes) and alert at 70% usage.
Migrating from ZooKeeper: follow the official migration guide node by node; do not reuse a ZooKeeper data directory for KRaft.