Installing Grafana 9.5 and Adding Data Sources

Grafana turns metrics and logs from many systems into dashboards and alerts. Install it once, connect it to your data sources, and the same interface can show Prometheus metrics from Kubernetes, CloudWatch metrics from AWS, and logs from Loki or Elasticsearch.

Versions in use on this date. Grafana OSS 9.5 (April 2023; version 10.0 is expected within weeks), Prometheus 2.44 and Ubuntu 22.04 LTS. Grafana publishes packages for Debian and Ubuntu at apt.grafana.com, and signs them with a key you install into /etc/apt/keyrings.

Architecture

flowchart TB
  USER[Engineers] -->|HTTPS 443| NGX[nginx reverse proxy<br/>TLS, rate limits]
  NGX -->|HTTP 3000 on localhost| G
  subgraph HOST[Grafana host - Ubuntu 22.04]
    G[grafana-server]
    DB[(SQLite grafana.db<br/>or PostgreSQL for HA)]
    PROV[Provisioning files<br/>datasources, dashboards, alerting]
    G --- DB
    PROV --> G
  end
  G -->|HTTP 9090| PROM[Prometheus]
  G -->|AWS API with instance role| CW[CloudWatch metrics and Logs]
  G --> LOKI[Loki or other sources]
  G -->|alerts| NOTIFY[Slack, email, PagerDuty]
  IDP[Identity provider via OAuth or SAML] -.-> G

Install

sudo apt update
sudo apt install -y apt-transport-https software-properties-common wget gnupg
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
apt-cache policy grafana | head -4
sudo apt install -y grafana
sudo systemctl enable --now grafana-server
systemctl status grafana-server --no-pager
curl -s http://localhost:3000/api/health

To pin to a known version, install grafana=9.5.* (use the exact version that apt-cache policy shows) and hold it during planned maintenance with sudo apt-mark hold grafana.

Important paths: configuration in /etc/grafana/grafana.ini, data in /var/lib/grafana (including grafana.db), logs in /var/log/grafana, and provisioning files in /etc/grafana/provisioning.

First login and basic configuration

Open http://<host>:3000 and sign in as admin with password admin, then change it immediately (the UI forces this). Better, set it before anyone can reach the port:

sudo grafana-cli admin reset-admin-password '<a-long-random-password>'

Edit /etc/grafana/grafana.ini (or use GF_ environment variables in a systemd override):

[server]
domain = grafana.example.com
root_url = https://grafana.example.com/
http_addr = 127.0.0.1
http_port = 3000

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false

[security]
cookie_secure = true
cookie_samesite = lax
disable_gravatar = true

Restart with sudo systemctl restart grafana-server. Binding to 127.0.0.1 means only the reverse proxy reaches it. Use nginx with a TLS certificate in front, passing Host, X-Forwarded-For and X-Forwarded-Proto, and support WebSocket upgrade for Grafana Live. Grafana’s own TLS can also be enabled directly if you do not run a proxy.

Connect Prometheus

Connections, Data sources, Add data source, Prometheus.

• URL: http://localhost:9090 (or the Prometheus service name).
• Scrape interval: set it to match Prometheus (for example 15s) so rate() panels are correct.
• Click Save & test; you should see “Successfully queried the Prometheus API”.

A first panel query for host CPU:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Import ready-made dashboards by ID under Dashboards, New, Import, for example 1860 (Node Exporter Full) once node_exporter is scraped. Review imported dashboards before relying on them, and keep your own copies in Git.

Connect CloudWatch

Add data source, CloudWatch. Choose authentication AWS SDK Default so the EC2 instance role is used, with default region ap-south-1. Never paste access keys if a role is possible. The role needs read access only:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["cloudwatch:ListMetrics", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmsForMetric"], "Resource": "*" },
    { "Effect": "Allow", "Action": ["logs:DescribeLogGroups", "logs:GetLogGroupFields", "logs:StartQuery", "logs:StopQuery", "logs:GetQueryResults", "logs:GetLogEvents"], "Resource": "*" },
    { "Effect": "Allow", "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions", "tag:GetResources"], "Resource": "*" }
  ]
}

For several accounts, use the assume role ARN field per data source or CloudWatch cross-account observability. Remember that CloudWatch API calls are billed, so avoid dashboards that auto-refresh every 5 seconds over wide ranges.

Provision rather than click

Files in /etc/grafana/provisioning/datasources/ are loaded at start-up, so the setup is repeatable:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    jsonData: { timeInterval: 15s }
  - name: CloudWatch
    type: cloudwatch
    jsonData: { authType: default, defaultRegion: ap-south-1 }

Secrets go in secureJsonData using $__env{VAR} references, not plain text in Git.

Secure and operate

1. Authentication: connect your identity provider with generic OAuth, Google, GitHub or SAML (SAML is an Enterprise feature); map groups to Grafana roles (Viewer, Editor, Admin) and keep the built-in admin for emergencies.
2. Organisations, folders and permissions: give teams their own folders; viewers do not need edit rights.
3. Backups: copy /var/lib/grafana/grafana.db while stopped or using SQLite’s online backup, plus provisioning files in Git. Move to PostgreSQL or MySQL if you need more than one Grafana instance.
4. Updates: Grafana ships frequent patch releases with security fixes; update on a schedule and read the changelog for breaking changes (for example deprecated AngularJS plugins).
5. Plugins: install only needed plugins (grafana-cli plugins install <id>), from trusted sources, and review plugin signatures.
6. Monitoring Grafana: scrape /metrics, and alert if the health endpoint fails.

Troubleshooting

Symptom	Likely cause
Data source test: “connection refused”	Wrong URL, Prometheus not listening, or firewall; try curl from the Grafana host
CloudWatch “AccessDenied” or empty regions	Instance role missing permissions, wrong region, or IMDSv2 hop limit in containers
Login loop behind proxy	root_url and forwarded headers do not match the public URL
Dashboard shows “No data”	Time range, wrong variable value, or metric names differ from the dashboard’s expectation
Panels slow	Queries too broad; reduce range, add $__rate_interval, use recording rules in Prometheus