Grafana 10: Alerting and Dashboards as Code

Grafana 10.0 (12 June 2023) continues the unified alerting model that became the default in version 9: alert rules are written against any data source, evaluated by Grafana, and routed with contact points and notification policies. Version 10 also continues the work to remove AngularJS-based plugins and panels, so check that your dashboards and plugins do not depend on them before upgrading.

What is worth doing as soon as you have more than a handful of dashboards and alerts is keeping them in Git. Hand-edited alerts and dashboards drift, vanish when a database is lost, and cannot be reviewed.

From files to a running Grafana

flowchart TB
  REPO[(Git repository<br/>provisioning and dashboards)] -->|pull request review| MAIN[main branch]
  MAIN --> CI[CI pipeline validates JSON and YAML]
  CI --> DEPLOY[Deploy to /etc/grafana/provisioning<br/>or Kubernetes ConfigMaps]
  DEPLOY --> G[Grafana loads at start and on reload]
  G --> DS[Data sources]
  G --> DASH[Dashboards in folders]
  G --> RULES[Alert rules]
  RULES --> NP[Notification policy tree<br/>matches labels]
  NP --> CP1[Contact point on-call Slack]
  NP --> CP2[Contact point email]
  NP --> CP3[Contact point PagerDuty]
  TF[Terraform grafana provider] -.->|alternative to files| G

The alerting model in words

An alert rule says: CPU above 85 percent for 10 minutes, labelled severity=warning, send to the on-call contact point. Pieces:

Piece	Role
Alert rule	Query, condition, evaluation interval, for duration, labels, annotations
Labels	Key-value tags used for routing (severity, team, service)
Notification policy	Tree that matches labels and chooses a contact point, grouping and repeat interval
Contact point	Where to send (Slack, email, PagerDuty, webhook)
Mute timing / silence	Scheduled or ad hoc suppression, for maintenance
States	Normal, Pending (condition met but within for), Alerting, NoData, Error

Build the first rule in the UI (Alerting, Alert rules, New alert rule) to see all the fields, then export it and keep it as code.

Provision data sources and dashboards

/etc/grafana/provisioning/datasources/prom.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

/etc/grafana/provisioning/dashboards/default.yaml:

apiVersion: 1
providers:
  - name: default
    folder: Ops
    type: file
    disableDeletion: true
    allowUiUpdates: false
    options: { path: /var/lib/grafana/dashboards, foldersFromFilesStructure: true }

Export a dashboard’s JSON from the UI (Share, Export, Export for sharing externally), commit it, and Grafana loads it at start-up and periodically. With allowUiUpdates: false, UI edits to provisioned dashboards cannot be saved, which forces changes through Git. Use variables (such as $instance, $namespace) and a fixed uid so one dashboard serves many hosts and links keep working.

Provision alerting

Rules, contact points and policies each have a provisioning file under /etc/grafana/provisioning/alerting/. A rule file (check the exact schema by exporting a rule from your instance, because fields grow between versions):

apiVersion: 1
groups:
  - orgId: 1
    name: node-alerts
    folder: Ops
    interval: 1m
    rules:
      - uid: node-cpu-high
        title: High CPU
        condition: C
        for: 10m
        noDataState: NoData
        execErrState: Error
        labels: { severity: warning, team: platform }
        annotations:
          summary: "CPU above 85 percent on {{ $labels.instance }}"
          runbook_url: https://wiki.example.com/runbooks/high-cpu
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: prometheus
            model:
              refId: A
              expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
          - refId: C
            datasourceUid: "__expr__"
            model:
              refId: C
              type: threshold
              expression: A
              conditions: [ { evaluator: { type: gt, params: [85] } } ]

Contact points and policies:

apiVersion: 1
contactPoints:
  - orgId: 1
    name: oncall-slack
    receivers:
      - uid: oncall-slack
        type: slack
        settings:
          url: $SLACK_WEBHOOK_URL        # supplied from the environment
          title: '{{ template "default.title" . }}'
policies:
  - orgId: 1
    receiver: oncall-slack
    group_by: ["alertname", "team"]
    routes:
      - receiver: oncall-slack
        object_matchers: [["severity", "=", "critical"]]
        repeat_interval: 1h

Keep secrets out of Git by using environment variables or your secret manager when rendering the files. Alternatively, manage the same objects with the Terraform Grafana provider (grafana_rule_group, grafana_contact_point, grafana_notification_policy, grafana_dashboard), which gives plans, state and drift detection, or with the HTTP provisioning API.

Good alerts

1. Alert on symptoms (error rate, latency, saturation), with a runbook link in the annotation.
2. Use for to ignore short spikes, and route by severity and team labels rather than per-alert special cases.
3. Decide what NoData means: for a service that should always emit metrics it is a problem; for a rate that is legitimately zero it is not.
4. Group notifications so one outage sends one message, not fifty.
5. Test the pipeline: Contact points, Test, and a deliberate firing rule in staging.

Upgrading to 10

• Read the breaking changes and deprecation notes for 10.0, particularly around AngularJS plugins and legacy alerting leftovers.
• Upgrade a staging copy first with a restored backup of grafana.db.
• Check that provisioned files still load (journalctl -u grafana-server | grep -i provisioning), and re-export one dashboard after the upgrade to see schema changes.
• Keep dashboards small and focused: fewer panels per dashboard load faster and are easier to review as JSON diffs.