Health Probes and Resource Requests

Kubernetes can only keep applications healthy if you tell it what healthy means (probes) and how much capacity each pod needs (requests and limits). Wrong values cause some of the most baffling production problems: pods killed in a restart loop during start-up, traffic sent to pods that are not ready, nodes that run out of memory, and applications that mysteriously run slowly.

Versions in use on this date. Kubernetes 1.21, where the startup probe is stable (since 1.20). kubectl top needs metrics-server (see the autoscaling post).

What each probe controls

flowchart TB
  START[Container starts] --> SP{Startup probe<br/>until it first succeeds}
  SP -->|fails beyond threshold| KILL1[Container killed and restarted]
  SP -->|succeeds| RUN[Running]
  RUN --> LP{Liveness probe}
  LP -->|keeps failing| KILL2[kubelet restarts the container]
  RUN --> RP{Readiness probe}
  RP -->|passes| IN[Pod in Service endpoints<br/>receives traffic]
  RP -->|fails| OUT[Removed from endpoints<br/>container keeps running]

Probe	Question	On failure	Use for
Startup	Has the app finished starting?	Restart after the budget is exhausted	Slow starters (JVM, migrations); disables the other probes until it passes
Readiness	Can it serve requests right now?	Taken out of load balancing, not restarted	Warm-up, dependency checks, overload, draining
Liveness	Is it stuck beyond recovery?	Container restarted	Deadlocks, hung event loops

Writing them well

containers:
  - name: api
    image: registry.example.com/shop/api:2.3.1
    ports: [{ name: http, containerPort: 8080 }]
    startupProbe:
      httpGet: { path: /healthz/started, port: http }
      periodSeconds: 5
      failureThreshold: 36            # up to 3 minutes to start
    readinessProbe:
      httpGet: { path: /healthz/ready, port: http }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    livenessProbe:
      httpGet: { path: /healthz/live, port: http }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3

Rules that prevent outages:

1. Liveness must be cheap and self-contained. Do not check the database in a liveness probe: when the database is down every pod restarts at once and the outage gets worse. Liveness answers only “is this process responsive?”.
2. Readiness may check dependencies the pod needs to serve (a cache or database connection), but be careful not to make every pod unready together when a shared dependency blips.
3. Use a startup probe instead of a long initialDelaySeconds on liveness.
4. Choose a different path per probe, and make the endpoints unauthenticated and fast.
5. Probe types: httpGet, tcpSocket, exec (runs a command; costly if frequent).
6. Pair with graceful shutdown (SIGTERM handling, preStop) so removal from endpoints and process exit do not race.

Requests and limits

resources:
  requests: { cpu: 250m, memory: 256Mi }     # what the scheduler reserves
  limits:   { memory: 512Mi }                # hard cap; CPU limit omitted deliberately

• Requests decide placement and the share of CPU the container is guaranteed under contention.
• Memory limit: exceeding it gets the container OOM-killed (Reason: OOMKilled, exit 137). Memory cannot be reclaimed by slowing a process down.
• CPU limit: a quota per 100 ms period; going over causes throttling (slow responses), not a kill. Many teams set CPU requests but no CPU limit for latency-sensitive services, accepting some noisy-neighbour risk; others set limits for fairness. Measure throttling (container_cpu_cfs_throttled_seconds_total in Prometheus) before deciding.
• Applications must respect limits: set JVM heap (-XX:MaxRAMPercentage=75) and Node heap sizes.

QoS classes

Class	Condition	Under node memory pressure
Guaranteed	Every container has requests equal to limits (CPU and memory)	Evicted last
Burstable	At least one request or limit set	Evicted in order of usage over requests
BestEffort	No requests or limits	Evicted first

Production services should never be BestEffort.

Choosing the numbers

kubectl top pods -n shop --containers
kubectl top nodes
kubectl describe node worker-1 | sed -n '/Allocated resources/,/Events/p'

Start from measurements: observe a week of real usage in Prometheus (p95 CPU, max memory), set the request near typical usage and the memory limit with 20 to 30% headroom over the observed peak. Revisit after each release. Vertical Pod Autoscaler in recommendation mode can suggest values.

Guardrails for a namespace:

apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: shop }
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default: { memory: 256Mi }
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: team-quota, namespace: shop }
spec:
  hard: { requests.cpu: "8", requests.memory: 16Gi, limits.memory: 32Gi, pods: "50" }

Troubleshooting

• Restart loop right after deploy: liveness fires before the app starts; add a startup probe. Check kubectl describe pod for Liveness probe failed.
• Pod Ready but errors: readiness path checks too little.
• OOMKilled: raise the memory limit or fix a leak; look at kubectl get pod -o jsonpath='{.status.containerStatuses[0].lastState}'.
• Pending with Insufficient cpu: requests exceed free capacity; right-size or add nodes.
• Evicted pods: node memory or ephemeral storage pressure; set ephemeral-storage requests and review BestEffort pods.