When a pod misbehaves, the status column tells you which part of the system to look at. Memorise the status-to-cause mapping and you will fix most problems in a few commands, without guessing.
Versions in use on this date. Kubernetes and kubectl 1.23. In 1.23, ephemeral containers and kubectl debug are beta and enabled by default, which makes debugging minimal images much easier.
Start from the status
flowchart TB
S[kubectl get pods] --> ST{STATUS}
ST -->|Pending| P1[Scheduling - describe pod Events]
ST -->|ContainerCreating| P2[Volumes, secrets, network setup]
ST -->|ImagePullBackOff or ErrImagePull| P3[Image name tag registry credentials]
ST -->|CreateContainerConfigError| P4[Missing ConfigMap or Secret key]
ST -->|CrashLoopBackOff| P5[App exits - logs --previous]
ST -->|Running but not Ready| P6[Readiness probe or app port]
ST -->|OOMKilled or Error 137| P7[Memory limit too low or leak]
ST -->|Evicted| P8[Node pressure disk memory]
ST -->|Terminating forever| P9[Finalizers or node gone]
ST -->|Completed| P10[Normal for Jobs]
First three commands, always
kubectl get pod web-abc -o wide # node, IP, restarts, age
kubectl describe pod web-abc # Events at the bottom: the most useful lines
kubectl logs web-abc --all-containers --previous # logs of the crashed attempt (omit --previous if running)
kubectl get events --sort-by=.lastTimestamp -n shop | tail -20Pending
The scheduler cannot place the pod. describe shows a line like 0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had taint {...} that the pod didn't tolerate.
- Insufficient cpu/memory: requests too large for any node, or the cluster is full.
kubectl describe nodes | grep -A5 "Allocated resources"; scale nodes (Cluster Autoscaler) or lower requests. - Taints and tolerations / node selectors / affinity: the pod demands labels or tolerations no node has.
- PVC not bound:
kubectl get pvc; wrong StorageClass, or a zonal volume in a different zone from available nodes. - ResourceQuota exceeded:
kubectl describe quota -n shop.
ImagePullBackOff
kubectl describe pod web-abc | grep -A3 -i "failed to pull"
kubectl get pod web-abc -o jsonpath='{.spec.containers[*].image}'
kubectl get secret regcred -n shop # imagePullSecrets present in the same namespace?Typos in the tag, a private registry without imagePullSecrets, expired ECR tokens on nodes without the correct IAM role, Docker Hub rate limits (toomanyrequests), or network egress blocked to the registry.
CrashLoopBackOff
The container starts, exits, and Kubernetes restarts it with increasing back-off.
kubectl logs web-abc --previous --tail=100
kubectl get pod web-abc -o jsonpath='{.status.containerStatuses[0].lastState}' # exit code and reason
kubectl describe pod web-abc | grep -A8 "Last State"| Exit code / reason | Meaning |
|---|---|
| 1 or 2 | Application error: read the log |
| 126 / 127 | Command not executable / not found (wrong command or image) |
137 (OOMKilled) |
Killed for memory; raise the limit or fix the leak |
| 139 | Segmentation fault |
| 143 | SIGTERM (normal stop) |
Also check configuration: missing environment variables, wrong database URL, permissions on mounted files (non-root user and read-only filesystem), and a liveness probe killing a slow-starting app (add a startup probe).
Running but not Ready / no traffic
kubectl describe pod web-abc | grep -A3 -i "readiness"
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl port-forward pod/web-abc 8080:8080 & curl -i localhost:8080/healthz
kubectl get svc web -o jsonpath='{.spec.selector}'; kubectl get pod web-abc --show-labelsWrong probe path or port, app listening on 127.0.0.1 only, Service selector not matching pod labels, a NetworkPolicy blocking the kubelet or the ingress.
Debug inside a minimal container
kubectl exec -it web-abc -- sh # if the image has a shell
kubectl debug -it web-abc --image=busybox:1.35 --target=web # ephemeral container sharing the process namespace
kubectl debug web-abc -it --copy-to=web-debug --container=web -- sh # copy of the pod with a different command
kubectl debug node/ip-10-0-1-5.ap-south-1.compute.internal -it --image=ubuntu:22.04 # shell on a nodeUse nslookup, curl, nc -zv from there to test DNS and connectivity exactly as the pod sees it. Delete debug copies afterwards.
Terminating that never ends
kubectl get pod web-abc -o jsonpath='{.metadata.finalizers}'
kubectl delete pod web-abc --grace-period=0 --force # last resort; may leave processes if the node is aliveA pod on an unreachable node stays Terminating until the node returns or is removed. Finalizers waiting for a controller (storage, custom resources) need the controller fixed rather than patched away.
Node-level causes
kubectl get nodes
kubectl describe node worker-3 | sed -n '/Conditions/,/Addresses/p' # MemoryPressure DiskPressure PIDPressure Ready
kubectl top nodes; kubectl top pods -A --sort-by=memory | headNotReady nodes point to kubelet, runtime or network problems: check kubelet logs (journalctl -u kubelet), container runtime status (crictl ps), disk space and the CNI.
Habit that saves time
Record what you see (status, events, exit code) before changing anything, change one thing at a time, and capture the fix in a runbook or an alert annotation so the next on-call engineer starts one step ahead.