mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

Archive · Kubernetes

Getting Ready for Dockershim Removal in 1.24

Kubernetes 1.24 removes dockershim. How to find nodes and workloads that depend on Docker Engine, migrate to containerd, and the cri-dockerd alternative.

4 min read

  • Kubernetes 1.23
  • containerd 1.6
  • Kubernetes 1.24
  • crictl
  • cri-dockerd

Kubernetes 1.24, scheduled for early May 2022 (it slipped from mid-April), removes dockershim, the built-in adapter that let the kubelet drive Docker Engine. Clusters whose nodes still use Docker Engine as the container runtime cannot simply upgrade to 1.24. This post is a migration plan; the background is in the Docker 20.10 post from December 2020.

Versions in use on this date. Kubernetes 1.23, containerd 1.6 (February 2022). Containers built with Docker are unaffected, since they are standard OCI images; only the node runtime and tools that depend on the Docker daemon change.

What depends on Docker today

flowchart TB
  K[kubelet] --> Q{Runtime endpoint}
  Q -->|before 1.24 with Docker| DS[dockershim in kubelet]
  DS --> DD[dockerd] --> CD1[containerd] --> R1[runc]
  Q -->|containerd| CD2[containerd CRI plugin] --> R2[runc]
  Q -->|CRI-O| CRIO[CRI-O] --> R3[runc or crun]
  Q -->|Docker kept via adapter| CRD[cri-dockerd by Mirantis] --> DD
  X1[Pods mounting docker.sock] -.->|break| DD
  X2[Node scripts using docker ps] -.->|use crictl| CD2

Step 1: find out where you stand

kubectl get nodes -o wide                      # CONTAINER-RUNTIME column
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.containerRuntimeVersion}{"\n"}{end}'

Managed services:

  • EKS: Amazon Linux 2 AMIs for Kubernetes 1.21 and older default to Docker; containerd is available through the bootstrap flag --container-runtime containerd. AWS has said containerd will become the default AMI runtime for newer Kubernetes versions, so check the EKS documentation for your version before relying on the default.
  • GKE: node images based on containerd are the default and Docker-based node images are being retired; check your release channel.
  • AKS: new node pools use containerd from Kubernetes 1.19 on.
  • Self-managed (kubeadm, kops, k3s, RKE): look at /var/run/dockershim.sock versus /run/containerd/containerd.sock in the kubelet flags, kubeadm-flags.env, or /var/lib/kubelet/config.yaml.

Step 2: find workloads that talk to Docker

kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.hostPath.path=="/var/run/docker.sock") | .metadata.namespace + "/" + .metadata.name'
kubectl get ds,deploy,sts -A -o yaml | grep -n -E 'docker\.sock|/var/lib/docker|docker\.io/library/docker' | head

Typical offenders and their fixes:

Workload Why it breaks Fix
CI agents building images with the host’s Docker No Docker socket on nodes Kaniko, BuildKit, Buildah, or build outside the cluster
Log shippers reading /var/lib/docker/containers Different log path and format (CRI) Update to the CRI log path /var/log/pods and format
Monitoring agents using the Docker API API not present Upgrade to versions supporting CRI/containerd
Security or backup tools needing the socket Same Vendor update or a containerd-aware version
Node scripts or runbooks using docker ps No Docker CLI crictl ps, crictl logs, crictl images

Step 3: migrate nodes to containerd

Do it node pool by node pool, like any OS upgrade.

On kubeadm clusters:

kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
sudo systemctl stop kubelet
sudo apt-get install -y containerd.io        # from Docker's repository, or distro package
sudo mkdir -p /etc/containerd && containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml     # match kubelet cgroup driver
sudo systemctl restart containerd
# point kubelet at containerd (kubeadm-flags.env or kubelet config): 
#   --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock
sudo systemctl start kubelet
kubectl uncordon worker-1
kubectl get node worker-1 -o wide
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

Details that cause trouble: the cgroup driver must match between kubelet and runtime (systemd on systemd hosts), registry mirrors and credentials move into containerd’s configuration (config.toml), images already pulled by Docker are not shared, and containerd uses a different storage path. For cloud node groups, easier: create a new node group with a containerd-based AMI, cordon and drain the old one, and delete it.

Alternative: keep Docker with cri-dockerd

If you must keep Docker Engine on nodes (for example for tooling), install cri-dockerd from Mirantis, an out-of-tree adapter, and point the kubelet at unix:///run/cri-dockerd.sock. It is a bridge, not a long-term plan: you still carry two runtimes and extra components.

Step 4: test and roll out

  1. Build a staging cluster or node pool on containerd; run the whole test suite, including log collection and monitoring.
  2. Verify image pulls from every registry you use (private registries, mirrors, imagePullSecrets).
  3. Roll production node pools gradually, watching pod restarts and node readiness.
  4. Only then upgrade the control plane to 1.24, after the removed-API check.

Habits to build

  • Use crictl aliases and learn its parity with Docker commands (crictl ps, crictl logs, crictl inspect, crictl pull).
  • Treat the container runtime as an independent upgrade you can do ahead of Kubernetes upgrades.
  • Keep a list of every component that needs host-level integration and test it with each Kubernetes release.