A Kafka broker that restarts every few seconds looks alarming, but CrashLoopBackOff is only Kubernetes saying “the container keeps exiting, so I am waiting longer between restarts”. The reason is always in the exit code or in the log of the previous run. This post gives a repeatable order of checks, and the fix for each common cause.
Versions in use on this date. Kafka 3.8 running in KRaft mode (no ZooKeeper; ZooKeeper mode is deprecated and is removed in Kafka 4.0) as a StatefulSet on Kubernetes 1.30. The same checks apply if the brokers are managed by an operator such as Strimzi, with different object names.
The method
flowchart TB
A[kubectl get pods<br/>CrashLoopBackOff] --> B[kubectl describe pod<br/>exit code and events]
B --> C{Exit code}
C -->|137 OOMKilled| M[Raise memory limit<br/>and heap]
C -->|1 with error in log| L[kubectl logs --previous]
C -->|Pending or Init error| P[PVC and scheduling events]
L --> X{Message}
X -->|Invalid cluster.id| ID[Fix meta.properties]
X -->|No space left| DS[Expand PVC or delete segments]
X -->|Corrupt index| LR[Log recovery or rebuild]
X -->|Listener error| LN[Fix advertised listeners]
Step 1: read what Kubernetes knows
kubectl get pods -n kafka -o wide
kubectl describe pod kafka-1 -n kafka | sed -n '/State:/,/Ready:/p;/Events:/,$p'
kubectl logs kafka-1 -n kafka --previous --tail=100--previous is the key flag: the current container has just started and has no useful log yet, while the previous one holds the error that killed it. In describe, look at Last State: Terminated: Reason and Exit Code tell you most of the story.
| Exit code / reason | Meaning |
|---|---|
137, OOMKilled |
The container exceeded its memory limit |
| 1 | The JVM or Kafka exited on an error; read the log |
| 143 | Killed by SIGTERM, often a failing liveness probe |
Error before start |
Volume or config problem; check events |
Cause 1: out of memory (137)
The JVM heap plus page cache plus metaspace must fit under the container limit. A broker with -Xmx of 4 GB needs a limit well above that.
kubectl set resources statefulset kafka -n kafka --limits=memory=8Gi --requests=memory=6Gi
kubectl set env statefulset kafka -n kafka KAFKA_HEAP_OPTS="-Xms4g -Xmx4g"Set the heap to about half of the limit, because Kafka relies on the operating system page cache for fast reads.
Cause 2: cluster ID or node ID mismatch
The log says Invalid cluster.id in: /var/lib/kafka/data/meta.properties or Node ID does not match. This happens when a volume from an old cluster is attached to a new one, or when the cluster was re-created with a new ID but old PVCs were kept.
kubectl exec kafka-0 -n kafka -- grep -E 'cluster.id|node.id' /var/lib/kafka/data/meta.propertiesCompare with the healthy brokers. The safe repair is to make the pod use the correct cluster ID (the CLUSTER_ID environment variable the formatting step uses) so it matches the other brokers’ value. If the volume holds nothing you need, delete only that broker’s PVC and let it re-create and resync from the other replicas. Never delete the PVCs of several brokers at once, or the partitions they held are gone.
Cause 3: disk full
The log shows No space left on device or KafkaStorageException.
kubectl exec kafka-1 -n kafka -- df -h /var/lib/kafka/data
kubectl patch pvc data-kafka-1 -n kafka -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'Expanding works only if the StorageClass has allowVolumeExpansion: true. A crash-looping broker cannot be exec’d into, so for a stuck pod use a temporary pod that mounts the same PVC to look at the folders, then reduce log.retention.hours or log.retention.bytes on the topics so old segments are deleted when it starts. The real fix is a retention policy and an alert at 75% disk.
Cause 4: wrong advertised listeners
The broker starts and exits at once, or clients connect and time out. In Kubernetes each broker must advertise its own stable DNS name:
advertised.listeners=PLAINTEXT://kafka-1.kafka-headless.kafka.svc.cluster.local:9092Derive it from the pod name ($(hostname)) in the start script so one StatefulSet template works for every broker. A headless service gives each pod its own DNS record.
Cause 5: corrupt log segments after a hard kill
After an unclean shutdown the broker rebuilds indexes in a “log recovery” phase. On a large disk this takes minutes, and a liveness probe that is too strict kills it again and again.
kubectl logs kafka-1 -n kafka --previous | grep -iE 'recover|corrupt|index'If recovery is just slow, raise the probe’s initialDelaySeconds and failureThreshold (or use a startup probe) and add more num.recovery.threads.per.data.dir. Truly corrupt segments can be checked with kafka-dump-log.sh --files <segment>; if a partition is replicated, delete the broken segment files and the broker re-fetches them from the leader.
After it is back
kubectl exec kafka-0 -n kafka -- kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
kubectl exec kafka-0 -n kafka -- kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groupsAn empty under-replicated list means every partition has caught up. Do not restart the next broker until it is empty, otherwise you can take a partition below its minimum in-sync replicas and stop producers.
Prevent the next one
Use a PodDisruptionBudget of maxUnavailable: 1, topologySpreadConstraints across nodes, replication.factor 3 with min.insync.replicas 2, a startup probe, and alerts on disk, memory and under-replicated partitions.