Docker Logging and Resource Limits

Two things take down Docker hosts more than anything else: a container that logs without limit and fills the disk, and a container that uses all the memory or CPU and starves its neighbours. Both are solved with settings you apply once, on the daemon and on each container.

Versions in use on this date. Docker Engine 20.10.3 (released 1 February 2021). Version 20.10 added dual logging, so docker logs keeps working even when you use a remote log driver, and it supports cgroup v2 on hosts that use it.

Where the limits apply

flowchart TB
  APP[Container process] -->|stdout and stderr| DRV[Docker log driver]
  DRV --> JF[json-file or local<br/>rotated on host disk]
  DRV --> RM[Remote driver<br/>syslog fluentd awslogs]
  JF --> DL[docker logs reads the files]
  CG[cgroups] --> MEM[memory limit]
  CG --> CPU[cpu shares and quota]
  CG --> PID[pids limit]
  APP -.-> CG
  MEM -->|exceeded| OOM[Process killed - exit 137<br/>OOMKilled true]
  JF -->|no rotation| FULL[Host disk fills]

Logging

The default json-file driver stores each container’s output in /var/lib/docker/containers/<id>/<id>-json.log and, by default, never rotates it.

Set daemon-wide defaults in /etc/docker/daemon.json:

{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}

sudo systemctl restart docker      # applies to NEW containers only; recreate existing ones
docker inspect -f '{{.HostConfig.LogConfig}}' web

Driver	Notes
json-file	Default; human-readable; set max-size and max-file
local	Compact binary format with built-in rotation (10 MB x 5 by default); Docker recommends it for local logging
journald	Logs go to the systemd journal
syslog, fluentd, gelf, awslogs	Ship to a central system; with 20.10 docker logs still works through dual logging

Per-container override:

docker run -d --name api --log-opt max-size=20m --log-opt max-file=3 myapp:1.0.0
docker logs --since 10m --tail 100 -f api

Guidelines: let applications log to stdout/stderr (not to files inside the container), log structured JSON, ship logs off the host with an agent for anything important, and never log secrets. Find large logs with sudo du -ah /var/lib/docker/containers | sort -rh | head.

Memory

docker run -d --name api --memory 512m --memory-reservation 384m --memory-swap 512m myapp:1.0.0

• --memory is the hard limit; a container that exceeds it is killed by the kernel (OOM) and exits with code 137.
• --memory-swap equal to --memory disables swap for the container; unset, the container may use as much swap as RAM.
• --memory-reservation is a soft target that the kernel enforces under host pressure.

docker inspect -f 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' api
dmesg -T | grep -i 'memory cgroup out of memory'

Size limits from measurements (docker stats over a typical day), then add headroom. Runtimes need configuration to respect limits: set Java heap with -XX:MaxRAMPercentage, Node with --max-old-space-size.

CPU

docker run -d --cpus 1.5 myapp:1.0.0                  # at most 1.5 cores worth of time
docker run -d --cpu-shares 512 myapp:1.0.0            # relative weight when CPUs are contended (default 1024)
docker run -d --cpuset-cpus 0-1 myapp:1.0.0           # pin to cores 0 and 1

--cpus is a hard cap (quota), --cpu-shares only matters under contention. A CPU limit that is too tight shows up as throttled latency, not as a crash.

Other safeguards

docker run -d --pids-limit 200 --ulimit nofile=65536:65536 --read-only --tmpfs /tmp --restart unless-stopped myapp:1.0.0
docker update --memory 1g --memory-swap 1g api        # change limits on a running container

• --pids-limit stops fork bombs and runaway thread creation.
• --read-only with a tmpfs for scratch space protects the image layers from changes.
• --restart on-failure:5 retries a few times, not forever, so crash loops become visible.

Monitoring usage

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}"
docker system df                      # images, containers, volumes, build cache
docker system prune --filter "until=168h"      # remove old unused objects (review first)

For fleets, export container metrics with cAdvisor or the Docker metrics endpoint, and alert on memory above 85% of limit, restart counts, and host disk usage.

Troubleshooting

• Host disk full: log files (du on /var/lib/docker/containers) or build cache (docker builder prune).
• Container keeps restarting with 137: raise the memory limit or fix a leak; check OOMKilled.
• Limits ignored: the kernel may lack swap accounting; check docker info warnings.
• Slow container with CPU quota: look at docker stats and the cgroup’s cpu.stat throttling counters.