Prometheus and Grafana with Docker Compose

Prometheus collects numeric metrics by pulling them from HTTP endpoints, stores them in a time-series database and evaluates alert rules; Grafana turns them into dashboards. A single Compose file gives you a complete monitoring stack for a server or a small team in minutes.

Versions in use on this date. Prometheus 2.25, node_exporter 1.1, Grafana 7.4 and Docker Compose 1.28. Pin versions as shown, because Grafana in particular changes UI and defaults between releases.

Stack layout

flowchart TB
  subgraph HOST[Docker host]
    NE[node_exporter<br/>host CPU memory disk  9100] 
    CA[cAdvisor<br/>container metrics  8080]
    APP[Your app /metrics  3000]
    PR[Prometheus  9090<br/>TSDB volume prom_data]
    GF[Grafana  3000<br/>volume grafana_data]
    AM[Alertmanager  9093]
  end
  PR -->|scrape every 15 s| NE
  PR -->|scrape| CA
  PR -->|scrape| APP
  GF -->|PromQL queries| PR
  PR -->|firing alerts| AM
  AM --> EM[Email or chat webhook]
  USER[You] -->|HTTPS via reverse proxy| GF

docker-compose.yml

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.25.0
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prom_data:/prometheus
    ports: ["127.0.0.1:9090:9090"]

  node-exporter:
    image: prom/node-exporter:v1.1.2
    restart: unless-stopped
    pid: host
    command: ["--path.rootfs=/host"]
    volumes: ["/:/host:ro,rslave"]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.37.5
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

  alertmanager:
    image: prom/alertmanager:v0.21.0
    restart: unless-stopped
    volumes: ["./alertmanager:/etc/alertmanager:ro"]

  grafana:
    image: grafana/grafana:7.4.3
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_USERS_ALLOW_SIGN_UP: "false"
    secrets: [grafana_admin]
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports: ["127.0.0.1:3000:3000"]

secrets:
  grafana_admin:
    file: ./secrets/grafana_admin.txt
volumes:
  prom_data:
  grafana_data:

Everything is bound to 127.0.0.1 except what a reverse proxy publishes; the exporters expose detailed host information and must not be open to the internet.

prometheus/prometheus.yml and a first alert

global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files: ["/etc/prometheus/alerts.yml"]
alerting:
  alertmanagers:
    - static_configs: [{ targets: ["alertmanager:9093"] }]
scrape_configs:
  - job_name: prometheus
    static_configs: [{ targets: ["localhost:9090"] }]
  - job_name: node
    static_configs: [{ targets: ["node-exporter:9100"] }]
  - job_name: cadvisor
    static_configs: [{ targets: ["cadvisor:8080"] }]

# prometheus/alerts.yml
groups:
  - name: host
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: { severity: critical }
        annotations: { summary: "{{ $labels.job }} on {{ $labels.instance }} is down" }
      - alert: DiskAlmostFull
        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes) < 0.15
        for: 10m
        labels: { severity: warning }
        annotations: { summary: "Less than 15% free on {{ $labels.mountpoint }}" }

Start and check:

mkdir -p secrets && openssl rand -base64 18 > secrets/grafana_admin.txt
docker-compose up -d
curl -s localhost:9090/-/ready
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

The /targets page in the Prometheus UI should show all jobs UP. Reload configuration without restart: curl -X POST localhost:9090/-/reload.

Grafana as code

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Import community dashboards by ID (for example Node Exporter Full, 1860) from the Grafana UI, then export the JSON into provisioning/dashboards so they survive rebuilds.

Queries worth knowing (PromQL)

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)       # CPU %
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes                           # memory available fraction
rate(container_cpu_usage_seconds_total{name!=""}[5m])                                 # CPU per container

Operations

• Retention and size: 30 days at 15-second scrapes for a few servers is a few GB; set --storage.tsdb.retention.size as a safety stop, and back up the volume only if you need history.
• Security: put Grafana behind HTTPS (reverse proxy), disable anonymous access, and protect Prometheus with a proxy and authentication if it ever leaves localhost.
• High availability: this stack is a single node; for critical environments run two Prometheus servers scraping the same targets, and send alerts to a redundant Alertmanager.
• Instrument your own app with a client library exposing /metrics (request counts, errors and duration histograms), then alert on error ratio and latency, which are the symptoms users feel.