Monitoring a Cluster with kube-prometheus-stack

Monitoring a cluster by hand-writing scrape configs does not scale as workloads come and go. The kube-prometheus-stack Helm chart installs a complete, opinionated monitoring system: the Prometheus Operator, Prometheus, Alertmanager, Grafana, node exporters, kube-state-metrics, dashboards and a large set of ready alert rules.

Versions in use on this date. The chart prometheus-community/kube-prometheus-stack (versions in the 30s at this time), installing Grafana 8.x and a Prometheus 2.3x server on Kubernetes 1.23. The chart’s major versions change often and sometimes need manual CRD steps, so always read the chart’s upgrade notes.

Components

flowchart TB
  subgraph MON[Namespace monitoring]
    OP[Prometheus Operator<br/>watches CRDs] -->|creates and configures| PROM[Prometheus StatefulSet<br/>PVC 50Gi retention 15d]
    OP --> AM[Alertmanager]
    SM[ServiceMonitor and PodMonitor] -->|which targets to scrape| OP
    PR[PrometheusRule<br/>alerts and recording rules] --> OP
    NE[node-exporter DaemonSet] -->|9100| PROM
    KSM[kube-state-metrics<br/>object state] --> PROM
    PROM -->|scrape| KUBELET[kubelet and cAdvisor]
    PROM -->|scrape| APP[Your app /metrics]
    PROM -->|fire alerts| AM
    GF[Grafana with dashboards sidecar] -->|PromQL| PROM
  end
  AM --> NOTIFY[Slack email PagerDuty]

The operator turns Kubernetes objects into Prometheus configuration. You describe what to monitor with a ServiceMonitor; the operator updates Prometheus without restarts.

Install

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update
helm show values prometheus-community/kube-prometheus-stack > values-default.yaml      # long: read it

A minimal values-prod.yaml:

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 45GB
    resources: { requests: { cpu: 500m, memory: 2Gi }, limits: { memory: 4Gi } }
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources: { requests: { storage: 50Gi } }
    serviceMonitorSelectorNilUsesHelmValues: false     # also pick up ServiceMonitors not labelled for this release
    ruleSelectorNilUsesHelmValues: false
alertmanager:
  config:
    route:
      receiver: slack
      group_by: [alertname, namespace]
      routes:
        - matchers: ['severity="critical"']
          receiver: slack
    receivers:
      - name: slack
        slack_configs:
          - channel: "#alerts"
            api_url_file: /etc/alertmanager/secrets/slack/url
            send_resolved: true
  alertmanagerSpec:
    secrets: [slack-webhook]
grafana:
  admin:
    existingSecret: grafana-admin
    userKey: admin-user
    passwordKey: admin-password
  persistence: { enabled: true, size: 10Gi }

kubectl create namespace monitoring
kubectl -n monitoring create secret generic grafana-admin --from-literal=admin-user=admin --from-literal=admin-password="$(openssl rand -base64 18)"
kubectl -n monitoring create secret generic slack-webhook --from-file=url=./slack-url.txt
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring -f values-prod.yaml --version <pinned-version> --atomic --timeout 10m
kubectl -n monitoring get pods
kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090

Open Prometheus Status, Targets and confirm kubelet, node-exporter, kube-state-metrics and the API server show UP. On managed Kubernetes (EKS, GKE, AKS) the control-plane components (scheduler, controller-manager, etcd) are not reachable, so disable their ServiceMonitors in the values (kubeScheduler.enabled: false, kubeControllerManager.enabled: false, kubeEtcd.enabled: false) to avoid permanently red targets and false alerts.

Monitor your own application

Expose /metrics (a client library for your language), give the Service a named port, then:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: shop-api, namespace: shop, labels: { release: monitoring } }
spec:
  selector: { matchLabels: { app: api } }
  namespaceSelector: { matchNames: [shop] }
  endpoints:
    - { port: http, path: /metrics, interval: 30s }
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: shop-api-alerts, namespace: shop, labels: { release: monitoring } }
spec:
  groups:
    - name: shop-api
      rules:
        - alert: ApiHighErrorRate
          expr: sum(rate(http_requests_total{job="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
          for: 10m
          labels: { severity: critical }
          annotations: { summary: "More than 5% of API requests fail", runbook_url: "https://wiki.example.com/runbooks/api-errors" }

The release: monitoring label matches the default selector; if your Prometheus ignores a ServiceMonitor, that label (or the NilUsesHelmValues settings) is the first thing to check.

What the included alerts cover

KubePodCrashLooping, KubeNodeNotReady, KubeDeploymentReplicasMismatch, NodeFilesystemSpaceFillingUp, KubePersistentVolumeFillingUp, CPUThrottlingHigh, PrometheusTargetMissing and many more, plus a Watchdog alert that always fires (route it to a dead-man’s-switch service so you know alerting itself works). Tune noisy ones by disabling groups in defaultRules.rules and silencing in Alertmanager.

Storage, retention and cost

Rough sizing: each active series costs about 1 to 2 KB of memory, and disk use is about 1 to 2 bytes per sample. A 100-node cluster can have a million series. Control growth by dropping high-cardinality labels (user IDs, request paths with IDs), raising scrape intervals, and setting retentionSize as a guard. For long-term storage and multi-cluster queries add Thanos, Mimir or a managed service (Amazon Managed Service for Prometheus).

Upgrades

The chart bundles CRDs, and Helm does not update CRDs on helm upgrade. For major chart upgrades, apply the new CRDs first (the chart README lists the commands, usually kubectl apply --server-side -f <crd urls>), then upgrade the release. Take a snapshot of the Prometheus volume or accept losing history, and test on a non-production cluster.