New Relic: NRQL Dashboards and Alerts

NRQL (New Relic Query Language) is the SQL-like language used across New Relic: in the query builder, dashboards and alert conditions. Once data is flowing from the agents in the previous post, NRQL turns it into answers and into alarms.

Telemetry to action

flowchart TB
  DATA[(Telemetry<br/>Transaction, TransactionError, SystemSample, Log)] --> NRQL[NRQL query]
  NRQL --> CHART[Chart in query builder]
  CHART --> DASH[Dashboard widgets]
  NRQL --> COND[Alert condition<br/>threshold on query result]
  COND --> POL[Alert policy]
  POL --> ISS[Issue opened<br/>incidents grouped]
  ISS --> WF[Workflow<br/>filter and route]
  WF --> SL[Slack channel]
  WF --> PD[PagerDuty]
  WF --> EM[Email]
  MUTE[Muting rule<br/>maintenance window] -.->|suppresses notifications| WF

NRQL you will use daily

-- CPU by host over the last hour
SELECT average(cpuPercent) FROM SystemSample FACET hostname SINCE 1 hour ago TIMESERIES

-- p95 latency for one service
SELECT percentile(duration, 95) FROM Transaction WHERE appName = 'shop-api' SINCE 1 day ago TIMESERIES

-- Top errors in the last hour
SELECT count(*) FROM TransactionError WHERE appName = 'shop-api' FACET error.message SINCE 1 hour ago LIMIT 10

-- Error rate as a percentage
SELECT percentage(count(*), WHERE error IS true) FROM Transaction WHERE appName = 'shop-api' TIMESERIES 5 minutes

-- Compare with yesterday
SELECT average(duration) FROM Transaction WHERE appName = 'shop-api' SINCE 1 hour ago COMPARE WITH 1 day ago TIMESERIES

Handy clauses: FACET (group by), TIMESERIES (chart over time), SINCE/UNTIL, COMPARE WITH, LIMIT, WHERE ... LIKE, filter(count(*), WHERE ...), histogram() and uniqueCount(). Use SELECT * FROM Transaction LIMIT 1 to discover attribute names, and SHOW EVENT TYPES for event tables.

Build a dashboard

A good service dashboard answers “is it healthy?” in ten seconds. Use one row per question:

Row	Widgets
Traffic and latency	Requests per minute, p50 and p95 duration
Errors	Error rate percent, top error messages, errors by host
Dependencies	Database and external call time, cache hit rate
Hosts	CPU, memory, disk and load for the service’s hosts
Deployments	Deployment markers on the time axis

Use dashboard variables (for example appName, environment) so one dashboard serves every service, and keep dashboards in Git with Terraform’s New Relic provider or the NerdGraph API once they matter, so they are reviewable and reproducible.

Create an alert condition from a query

1. Alerts, Alert conditions, New alert condition, Write your own query.
2. Query: SELECT percentage(count(*), WHERE error IS true) FROM Transaction WHERE appName = 'shop-api-production'
3. Window and thresholds: aggregation window 1 minute, critical when above 5 percent for at least 5 minutes, warning above 2 percent for 10 minutes.
4. Configure signal loss: open a violation if no data arrives for 10 minutes (an agent that stops reporting is worth knowing about) and use gap filling with a static value (0) for rate queries where “no data” means “no errors”.
5. Put the condition in a policy (for example shop-api-production), and set issue creation to group related incidents.
6. Create a workflow that matches the policy and sends notifications to a destination (Slack channel, PagerDuty service, email). Test the destination before relying on it.

-- Hosts: disk nearly full
SELECT max(diskUsedPercent) FROM StorageSample FACET hostname, mountPoint WHERE mountPoint = '/'
-- Alert above 85 for 10 minutes

-- Synthetics-like check of absence: no traffic at all
SELECT count(*) FROM Transaction WHERE appName = 'shop-api-production'
-- Alert below 1 for 10 minutes (with signal loss enabled)

Alert quality rules

• Alert on symptoms users feel (errors, latency, availability) first; use causes (CPU, disk) as lower-urgency warnings.
• Every page should be actionable and link to a runbook in the condition’s description.
• Prefer sustained thresholds (“for at least 5 minutes”) over single spikes; avoid flapping.
• Use separate policies per service and environment so production pages and staging only notifies a channel.
• Track alert fatigue: review monthly which conditions fired most and tune or delete the noisy ones.
• Add a muting rule for planned maintenance and deployments, scoped narrowly (the policy and time window), so deployments do not page anyone.

Cost and housekeeping

NRQL queries over long ranges and many facets use more query resources; keep dashboards efficient and set sensible default time ranges. Custom events, metrics and logs count towards ingest, so alert queries based on them depend on that data staying cheap. Review conditions when a service is renamed, since an appName mismatch silently stops alerting, which signal-loss detection will help you catch.