StatefulSets for Databases and Queues

Deployments treat pods as interchangeable. Databases, message brokers and clustered caches need the opposite: each member has an identity, its own disk, and a predictable start-up order. A StatefulSet provides stable names (db-0, db-1), stable DNS, one PersistentVolumeClaim per pod and ordered, controlled updates.

Versions in use on this date. Kubernetes 1.23 (7 December 2021) and PostgreSQL 14 (September 2021). The StatefulSet feature that retains or deletes claims automatically when a StatefulSet is removed, persistentVolumeClaimRetentionPolicy, is alpha in 1.23; by default claims are kept when you scale down or delete.

What the StatefulSet gives you

flowchart TB
  HS[Headless Service db<br/>clusterIP None] -->|DNS per pod| P0
  subgraph SS[StatefulSet db, 3 replicas]
    P0[db-0 primary<br/>starts first] --> V0[(PVC data-db-0)]
    P1[db-1 replica<br/>starts after db-0 Ready] --> V1[(PVC data-db-1)]
    P2[db-2 replica<br/>starts after db-1 Ready] --> V2[(PVC data-db-2)]
  end
  P0 -->|streaming replication| P1
  P0 -->|streaming replication| P2
  APP[Application] -->|db-rw service selects primary| P0
  APP2[Reporting] -->|db-ro service selects replicas| P1
  APP2 --> P2
  V0 -.->|EBS in one AZ| AZ[Availability Zone]

Guarantee	Meaning
Stable network identity	Pod db-0 keeps its name, and DNS db-0.db.shop.svc.cluster.local through headless Service db
Stable storage	volumeClaimTemplates create data-db-0, data-db-1, …; a rescheduled pod gets its own disk back
Ordered start, scale-down and update	By default db-0 first, then db-1; reverse on scale-down (podManagementPolicy: OrderedReady)
Controlled rolling updates	One pod at a time, highest ordinal first; partition allows canary-style updates

What it does not give you: replication, failover, backups or leader election. Those come from the database software or from an operator.

A sketch of a StatefulSet

apiVersion: v1
kind: Service
metadata: { name: db, namespace: shop }
spec:
  clusterIP: None
  selector: { app: db }
  ports: [{ name: pg, port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db, namespace: shop }
spec:
  serviceName: db
  replicas: 3
  selector: { matchLabels: { app: db } }
  updateStrategy:
    type: RollingUpdate
    rollingUpdate: { partition: 0 }
  template:
    metadata: { labels: { app: db } }
    spec:
      terminationGracePeriodSeconds: 120
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector: { matchLabels: { app: db } }
              topologyKey: topology.kubernetes.io/zone
      containers:
        - name: postgres
          image: postgres:14.1
          ports: [{ containerPort: 5432, name: pg }]
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
            - name: POSTGRES_PASSWORD
              valueFrom: { secretKeyRef: { name: db-credentials, key: password } }
          readinessProbe:
            exec: { command: ["pg_isready", "-U", "postgres"] }
            periodSeconds: 10
          resources:
            requests: { cpu: "1", memory: 2Gi }
            limits: { memory: 2Gi }
          volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 100Gi } }

This only starts three independent PostgreSQL servers. Making db-1 and db-2 replicas of db-0 needs init scripts or an operator that runs pg_basebackup, configures streaming replication and promotes a replica on failure. That is why most teams use an operator (Zalando Postgres Operator, Crunchy PGO, Percona, or a managed service) instead of writing it by hand.

Behaviour you must understand

• Storage is zonal. An EBS volume lives in one Availability Zone. If that zone fails, the pod cannot start elsewhere; the replica in another zone must take over (database-level failover).
• Scaling down keeps the data. kubectl scale statefulset db --replicas=2 leaves data-db-2 in place; delete it manually when you are certain.
• Deleting the StatefulSet does not delete claims or stop pods gracefully if you use --cascade=orphan; for orderly shutdown, scale to 0 first.
• Rolling update stops on a failing pod. If the new db-2 never becomes Ready, the controller waits; you may need to fix and delete the pod manually.
• Graceful shutdown matters: give terminationGracePeriodSeconds time to checkpoint, and use a preStop hook where the database needs it.
• Updates: use partition: 2 to update only db-2 first, verify, then lower the partition to roll the rest.

Protect and operate

kubectl -n shop get sts,pods,pvc -l app=db
kubectl -n shop rollout status sts/db
kubectl -n shop patch sts db -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl -n shop exec db-0 -- psql -U postgres -c "select client_addr,state,sync_state from pg_stat_replication;"

1. Add a PodDisruptionBudget (maxUnavailable: 1) so node drains cannot take down two members at once.
2. Use topology spread or anti-affinity across zones and nodes.
3. Back up outside the cluster: logical dumps (pg_dump), physical backups with WAL archiving to object storage (pgBackRest or WAL-G), and VolumeSnapshots. Test restores.
4. Set requests equal to limits for memory, giving the pod Guaranteed-like behaviour under pressure.
5. Monitor disk usage per volume, replication lag, and restarts.

When to use a StatefulSet, and when not to

Good fit: Kafka, ZooKeeper, Elasticsearch, etcd, Redis in cluster mode, and databases managed by an operator. For critical relational data, weigh the cost of running it yourself against a managed service such as Amazon RDS or Aurora: you gain backups, patching and Multi-AZ failover without building them. Kubernetes can run databases, but it does not remove the need for database operations skills.