mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Strimzi: Running Apache Kafka on Kubernetes, with CloudEvents for Portable Events

CNCF Incubating. An operator that deploys and manages Apache Kafka clusters, topics, users and connectors as Kubernetes resources; covered here with CloudEvents, the graduated CNCF standard for describing event data.

8 min read

CNCF Incubating

  • Strimzi
  • Apache Kafka
  • CloudEvents
  • Kubernetes
  • Cruise Control
CNCF project page

Strimzi is a Kubernetes operator for Apache Kafka. It turns a Kafka cluster, its topics, users, Kafka Connect deployments and MirrorMaker replication into Kubernetes custom resources, and handles installation, rolling upgrades, certificates, scaling and configuration changes. It addresses the operational weight of running Kafka on virtual machines by hand, where broker upgrades, TLS rotation and partition rebalancing are slow and error-prone. Red Hat started the project in 2017; it was accepted into the CNCF Sandbox on August 28, 2019 and moved to Incubating on February 8, 2024. This brief also covers CloudEvents, the CNCF Graduated specification (graduated January 25, 2024) for describing event data in a common way, which pairs naturally with Kafka-based event platforms.

CNCF daily · Sprint 2, day 2 · Streaming & Messaging

At a glance
Category App Definition and Development › Streaming & Messaging
CNCF status Incubating. Sandbox August 28, 2019; Incubating February 8, 2024
Written in Java
License Apache 2.0
Current release Strimzi 1.3.0, supporting Kafka 4.2.0 to 4.3.1 and tested on Kubernetes 1.32 to 1.37; since 1.0 (early 2026) the only supported CRD API is kafka.strimzi.io/v1
Metadata mode KRaft only (Kafka 4 removed ZooKeeper)
Related standard CloudEvents 1.0 specification, CNCF Graduated, with SDKs in nine languages and a Kafka protocol binding

Architecture

flowchart TB
  K[Kafka and KafkaNodePool resources<br/>listeners, config, version] -.->|desired state| CO
  subgraph OPS[Strimzi operators]
    CO[Cluster Operator]
    EO[Entity Operator<br/>Topic Operator and User Operator]
    CO -->|manages| EO
  end
  CO ==>|reconciles| PODS[Kafka pods<br/>KRaft quorum and brokers]
  EO -->|KafkaTopic and KafkaUser ACLs| PODS
  CO -.->|optional| OPT[KafkaConnect, MirrorMaker 2<br/>and Cruise Control]
  OPT -->|connect, mirror, rebalance| PODS
  PROD[Producers emit<br/>CloudEvents] ==>|publish| PODS
  PODS ==>|consume| CONS[Consumers read<br/>CloudEvents headers]
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef ctrl fill:#ede7f6,stroke:#5e35b1,stroke-width:1.5px,color:#1d1c1a
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef store fill:#fff3e0,stroke:#e65100,stroke-width:1.5px,color:#1d1c1a
  classDef sec fill:#fce4ec,stroke:#ad1457,stroke-width:1.5px,color:#1d1c1a
  classDef obs fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px,color:#1d1c1a
  class K,OPT ctrl
  class CO,EO core
  class PODS data
  class PROD,CONS actor
  style OPS fill:#0f766e14,stroke:#0f766e,stroke-width:2px

Key: teal blocks are the project itself, blue are workloads and proxies, purple are control and configuration, amber are storage, red are security and identity, green are observability, grey are external actors. Thick arrows show the primary path, thin solid arrows other data flows, dashed arrows control, discovery and telemetry.

Component Responsibility
Cluster Operator Deploys and upgrades Kafka, node pools, Connect, MirrorMaker 2 and Cruise Control; manages certificates and rolling restarts in a safe order.
KafkaNodePool Groups of Kafka nodes with their own roles (controller, broker or both), replica count and storage, allowing separate pools per role or zone.
Topic Operator Keeps Kafka topics in sync with KafkaTopic resources, so topics are declared and reviewed in Git.
User Operator Creates KafkaUser credentials (mTLS or SCRAM) and their ACLs and quotas, stored as Kubernetes Secrets.
Cruise Control Rebalances partitions across brokers through KafkaRebalance resources, including when scaling up or down.
Drain Cleaner Optional; intercepts node drains so brokers restart in an order that keeps partitions available.

Design principle. Everything is declarative. Operators reconcile Kafka towards the custom resources, so the cluster, topics and users are versioned YAML that a GitOps controller can apply, and day-two operations such as upgrades or certificate rotation become resource changes rather than runbooks.

CloudEvents in an event platform

Kafka moves bytes; it does not define what an event means. CloudEvents standardises the envelope: a small set of attributes every event carries, independent of the transport.

Attribute Example Purpose
specversion 1.0 Spec version
id a1b2c3 Unique per source, for de-duplication
source /shop/orders Who produced it
type com.acme.order.created What happened, used for routing
time, datacontenttype, subject optional Context for consumers

In Kafka’s binary content mode, attributes travel as record headers (ce_specversion, ce_id, ce_source, ce_type) and the payload stays unchanged; in structured mode the whole event is one JSON document. Using CloudEvents means routing, filtering, tracing and schema handling work the same whether an event crosses Kafka, HTTP or a cloud event bus such as Azure Event Grid or Google Cloud Eventarc.

Production reference design

A typical deployment on EKS, GKE, AKS or on-premises Kubernetes runs three dual-role nodes (or separate controller and broker pools for larger clusters) spread across zones.

  1. Install the operator: 
    kubectl create namespace kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
    In production, pin a specific release (or use the Helm chart with a fixed version) and upgrade deliberately.
  2. Declare the cluster with a node pool and the v1 API: 
    apiVersion: kafka.strimzi.io/v1
kind: KafkaNodePool
metadata:
  name: dual-role
  namespace: kafka
  labels:
    strimzi.io/cluster: events
spec:
  replicas: 3
  roles: [controller, broker]
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 200Gi
        kraftMetadata: shared
        deleteClaim: false
---
apiVersion: kafka.strimzi.io/v1
kind: Kafka
metadata:
  name: events
  namespace: kafka
spec:
  kafka:
    version: 4.3.1
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
    authorization:
      type: simple
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
  name: orders
  namespace: kafka
  labels:
    strimzi.io/cluster: events
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 604800000
    Use a Kafka version that your Strimzi release lists as supported (4.2.x or 4.3.x for Strimzi 1.3).
  3. Create application identities with KafkaUser resources (mTLS) and ACLs limited to the topics each service needs.
  4. Expose externally only when required, through loadbalancer, ingress or Gateway API listener types with TLS.
  5. Monitor with the Prometheus metrics configuration and the Strimzi Grafana dashboards; add Cruise Control for rebalancing.
  6. Adopt CloudEvents in producers through the official SDKs (Java, Go, JavaScript, Python and others), publishing in binary mode so existing consumers keep reading plain payloads.

Typical uses are event-driven microservices, change data capture with Kafka Connect and Debezium, log and metrics pipelines, and cross-region replication with MirrorMaker 2.

Operational considerations

  • Scaling: add brokers by increasing node pool replicas, then run a KafkaRebalance so existing partitions move; partitions do not move by themselves.
  • High availability: spread pools across zones with rack awareness, use replication factor 3 with min.insync.replicas: 2, and producers with acks=all. Install Drain Cleaner so node maintenance does not take out several replicas at once.
  • Security: keep TLS and authentication on every listener, use simple authorization or OAuth with Keycloak, and let the operator rotate cluster and client CAs.
  • Cost and sizing: Kafka is storage- and network-heavy; size disks for retention multiplied by replication, prefer provisioned-throughput volumes, and watch cross-zone traffic charges in the cloud.
  • Common pitfalls: upgrading Strimzi to 1.0 without first converting resources to the v1 API; letting topics be created outside the Topic Operator; and skipping Kafka version steps that a Strimzi release supports. Read the upgrade notes for each release.

Adoption

The CNCF incubation announcement named Axual, Atruvia, Decathlon, LittleHorse and SBB among Strimzi’s public adopters, and more are listed in the project’s ADOPTERS file. Commercial distributions build on it, including Red Hat’s Kafka offering for OpenShift and Cloudera’s Streams Messaging Operator. CloudEvents is used by CNCF projects such as Argo, Knative, Falco, Harbor and Serverless Workflow, and by services including Azure Event Grid, Google Cloud Eventarc, Alibaba Cloud EventBridge, Adobe I/O Events and IBM Cloud Code Engine, as cited in its graduation announcement.

Alternatives

Solution Model Best suited to
Amazon MSK, Confluent Cloud, Aiven Managed Kafka Teams that prefer to buy Kafka operations
Confluent for Kubernetes Commercial operator with Confluent Platform Organisations on Confluent Platform licences
Redpanda Kafka-compatible broker with its own operator Lower-latency, single-binary deployments
NATS JetStream (CNCF) Lightweight messaging and streams Simpler messaging without Kafka’s ecosystem
Apache Pulsar Separate compute and storage layers Multi-tenant, geo-replicated messaging

Strengths and limitations

Strengths Limitations
Kafka, topics, users and connectors as declarative resources Kafka itself still needs capacity planning and expertise
Safe rolling upgrades and automatic certificate management Major API changes (v1 in 1.0) require planned migrations
Node pools, Cruise Control and Gateway API listeners Storage performance on Kubernetes needs careful choice
Vendor-neutral and widely packaged by vendors Rebalancing is explicit and must be operated
Pairs cleanly with CloudEvents for portable event metadata CloudEvents adds attribute discipline that teams must enforce

Recommendation

Adopt Strimzi for teams that run Kafka on Kubernetes and want it managed declaratively alongside their applications; start with three dual-role nodes across zones, mTLS users and GitOps-managed topics, and add Cruise Control before the first scale-out. Adopt CloudEvents as the event envelope for new event-driven services, using binary mode on Kafka so it can be introduced without breaking existing consumers. Teams without Kafka operations experience should compare a managed service first.

References: Strimzi · Strimzi downloads and supported versions · Strimzi blog · CNCF project page: Strimzi · Strimzi joins the CNCF Incubator · Strimzi documentation · CloudEvents graduation announcement · CloudEvents specification

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.