Back Up etcd Before You Upgrade

On a self-managed (kubeadm) cluster, etcd holds everything: every Deployment, Secret, ConfigMap, RBAC rule and PVC definition. Lose etcd without a backup and the cluster is gone, even though the workloads’ disks may still exist. Before any upgrade, take a snapshot, prove it is readable, and keep a rollback plan.

Versions in use on this date. kubeadm and Kubernetes 1.23 with etcd 3.5. Note: etcd 3.5.0 through 3.5.2 have a known data-inconsistency bug that can occur after an unclean shutdown under certain conditions; the project has advised running 3.5.3 or newer once released (a fix is imminent), and Kubernetes 1.24 will ship with it. Until then, avoid unclean power loss on control-plane nodes, take frequent snapshots, and check the etcd release announcements. Kubernetes 1.24 itself is due in early May and removes dockershim (see the next post).

The upgrade sequence

flowchart TB
  A[Read release notes and removed APIs] --> B[Snapshot etcd and copy pki and manifests off the node]
  B --> C[Verify snapshot with etcdctl snapshot status]
  C --> D[Upgrade kubeadm on first control plane node]
  D --> E[kubeadm upgrade plan then apply]
  E --> F[Drain node, upgrade kubelet and kubectl, restart, uncordon]
  F --> G[Other control plane nodes - kubeadm upgrade node]
  G --> H[Workers one at a time - drain, kubeadm upgrade node, kubelet, uncordon]
  H --> I[Smoke tests and watch metrics]
  I -->|problem| R[Rollback - restore etcd snapshot and previous binaries]

Upgrade one minor version at a time (1.22 to 1.23, never 1.21 to 1.23), control plane first, then workers. Version skew rules allow kubelets to be up to two minor versions older than the API server, but stay close.

1. Snapshot etcd

On a kubeadm control plane node, etcd runs as a static pod with certificates under /etc/kubernetes/pki/etcd:

sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
sudo ETCDCTL_API=3 etcdutl snapshot status /var/backups/etcd-2022-04-04-1010.db -w table     # etcdctl snapshot status on older clients

If etcdctl is not installed on the host, run it in the etcd image: kubectl -n kube-system exec etcd-cp1 -- etcdctl ..., or crictl exec. Check the snapshot size and revision, and copy it off the node (object storage, another host):

sudo tar czf /var/backups/k8s-config-$(date +%F).tgz /etc/kubernetes /var/lib/kubelet/config.yaml
aws s3 cp /var/backups/ s3://acme-k8s-backups/prod/ --recursive --sse AES256 --exclude "*" --include "etcd-*" --include "k8s-config-*"

Take snapshots on a schedule (cron, a CronJob or Velero), not only before upgrades, and alert if the newest one is older than a day. Snapshots contain Secrets in plain form unless you enabled encryption at rest, so protect them like credentials.

2. Upgrade the first control plane node

# find the target version
apt-cache madison kubeadm | head
sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.23.5-00 && sudo apt-mark hold kubeadm
kubeadm version
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.23.5
kubectl drain cp1 --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.23.5-00 kubectl=1.23.5-00 && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon cp1
kubectl get nodes

kubeadm upgrade apply backs up manifests under /etc/kubernetes/tmp and upgrades API server, controller-manager, scheduler, etcd (if managed by kubeadm) and add-ons. On additional control plane nodes run sudo kubeadm upgrade node instead.

3. Workers, one at a time

kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
# on worker-1: upgrade kubeadm, then
sudo kubeadm upgrade node
# upgrade kubelet and kubectl packages, restart kubelet
kubectl uncordon worker-1
kubectl get pods -A -o wide | grep -v Running | grep -v Completed

PodDisruptionBudgets can block a drain; read the error, do not force it blindly. Make sure replicas exist elsewhere before draining nodes that hold single-replica or local-storage workloads.

4. Certificates

kubeadm certificates expire after one year, and upgrades renew them. Check anyway:

sudo kubeadm certs check-expiration
sudo kubeadm certs renew all        # then restart the control plane static pods

Restore (rollback) outline

If the control plane is broken beyond a simple fix:

1. Stop the API server and etcd (move the static pod manifests out of /etc/kubernetes/manifests).
2. Restore: etcdutl snapshot restore /var/backups/etcd-....db --data-dir /var/lib/etcd-restored.
3. Point the etcd manifest’s data directory at the restored path, return the manifests, and wait for the API.
4. Reinstall the previous version of kubeadm, kubelet and kubectl on the nodes if you downgraded.

Rehearse the restore on a throwaway cluster. A snapshot you have never restored is only a hope. Everything created after the snapshot is lost, and running pods are reconciled against the restored state.

Checklist

• Backups restorable, off the node, and recent
• Removed API check passed; add-ons (CNI, CoreDNS, ingress) support the target version
• Maintenance window and rollback criteria agreed
• PDBs and replica counts reviewed
• Monitoring watching API latency, node readiness and pod restarts during the upgrade