Kafka 3.7: Partitions, Replication, Consumer Groups and Lag

Most Kafka incidents come from misunderstanding four ideas: partitions, replication, acknowledgements and consumer groups. This post explains each with the settings that matter on Kafka 3.7 (released 27 February 2024), running in KRaft mode on three brokers.

How data and consumers are laid out

flowchart TB
  PR[Producer<br/>acks all idempotent] --> T
  subgraph T[Topic orders - 3 partitions, replication factor 3]
    P0[Partition 0<br/>leader on broker 1]
    P1[Partition 1<br/>leader on broker 2]
    P2[Partition 2<br/>leader on broker 3]
  end
  P0 -. follower copies .-> B2F[Brokers 2 and 3]
  P1 -. follower copies .-> B1F[Brokers 1 and 3]
  P2 -. follower copies .-> B3F[Brokers 1 and 2]
  subgraph G[Consumer group billing]
    C1[Consumer A] 
    C2[Consumer B]
    C3[Consumer C]
  end
  P0 --> C1
  P1 --> C2
  P2 --> C3
  G -->|commit offsets| OFF[Internal topic __consumer_offsets]

• A partition is an ordered log and the unit of parallelism. Ordering is guaranteed only within a partition; messages with the same key go to the same partition.
• Each partition has one leader (handles reads and writes) and follower replicas. The in-sync replica set (ISR) lists the followers that are caught up.
• Within one consumer group, each partition is read by at most one consumer, so more consumers than partitions leaves some idle. Different groups read independently.

Replication and durability settings

Setting	Recommended	Why
replication.factor (topic)	3	Survive a broker failure
min.insync.replicas	2	Writes need at least 2 copies
Producer acks	all	Wait for the ISR, not just the leader
enable.idempotence	true (default since Kafka 3.0)	No duplicates from producer retries
unclean.leader.election.enable	false	Never promote an out-of-date replica

With replication factor 3 and min.insync.replicas=2, you can lose one broker and keep both reading and writing; if two fail, writes are refused (a deliberate choice for consistency over availability). Create a topic with these guarantees:

bin/kafka-topics.sh --bootstrap-server k1:9092 --create --topic orders \
  --partitions 6 --replication-factor 3 --config min.insync.replicas=2 --config retention.ms=604800000
bin/kafka-topics.sh --bootstrap-server k1:9092 --describe --topic orders

--describe shows Leader, Replicas and Isr for each partition; Isr shorter than Replicas means a follower is behind or down. Choose partition count with growth in mind: partitions can be added later but never removed, and adding them changes key-to-partition mapping.

Producer configuration

acks=all
enable.idempotence=true
compression.type=lz4
linger.ms=10
batch.size=65536
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5

linger.ms and batch.size trade a little latency for throughput. For exactly-once pipelines across topics, use transactions (transactional.id) and consumers with isolation.level=read_committed.

Consumer configuration and rebalancing

group.id=billing
enable.auto.commit=false
auto.offset.reset=earliest
max.poll.records=500
max.poll.interval.ms=300000
session.timeout.ms=45000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id=billing-1

Disable auto-commit and commit offsets after processing, so a crash causes a re-read (at-least-once), not data loss; make processing idempotent. The cooperative sticky assignor moves only the partitions that must move during a rebalance, and a static group.instance.id lets a restarting consumer rejoin without triggering one. In Kafka 3.7 the new KIP-848 group protocol is an early-access preview, so do not rely on it for production yet. A consumer that takes longer than max.poll.interval.ms between polls is kicked out of the group, which is the usual cause of endless rebalances.

Inspecting groups and lag

bin/kafka-consumer-groups.sh --bootstrap-server k1:9092 --list
bin/kafka-consumer-groups.sh --bootstrap-server k1:9092 --describe --group billing

The output shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition. Steadily growing lag means consumers are too slow; alert on lag trends and not single spikes. Export metrics with a JMX exporter or Kafka Exporter into Prometheus, and watch under-replicated partitions, offline partitions and request latency.

Resetting offsets (the group must be stopped), always previewing first:

bin/kafka-consumer-groups.sh --bootstrap-server k1:9092 --group billing --topic orders \
  --reset-offsets --to-datetime 2024-06-17T00:00:00.000 --dry-run
# repeat with --execute once the plan looks right

Failure modes

• Broker down: leaders move to ISR followers within seconds; restart the broker and wait until kafka-topics.sh --describe --under-replicated-partitions is empty before touching the next one (rolling restarts).
• Disk full on one broker: it stops accepting writes; add disks or move partitions with kafka-reassign-partitions.sh.
• Hot partition: a skewed key sends most traffic to one partition; change the key or add a salt.