CloudWatch Logs and Logs Insights

Logs on individual servers vanish when an instance is replaced and are painful to search across a fleet. CloudWatch Logs collects them centrally, keeps them for as long as you choose, and Logs Insights lets you query them with a purpose-built language, so a question like “which URLs returned errors in the last hour?” takes seconds.

State of AWS on this date. Logs Insights has been generally available since 2018. Log events are charged for ingestion and storage; Insights charges per GB scanned, so narrow the time range and log groups before running queries. The unified CloudWatch agent (previous post) ships files from EC2; Lambda, API Gateway, ECS (awslogs driver) and RDS publish to log groups natively.

Flow of a log line

flowchart TB
  APP[Application writes JSON lines] --> F[Log file or stdout]
  F -->|CloudWatch agent or awslogs driver| LG[Log group web/app<br/>retention 30 days]
  LG --> LS1[Log stream per instance]
  LG --> MF[Metric filter<br/>count ERROR lines]
  MF --> AL[Alarm and SNS]
  LG --> INS[Logs Insights query]
  LG -->|subscription filter| KF[Kinesis or Lambda<br/>to S3 or analytics]
  LG -->|export task| S3[S3 archive]

Terms: a log group holds streams with shared retention and permissions (usually one per application); a log stream is one source (an instance or container); a log event is a timestamped record.

Collect logs

Using the CloudWatch agent file configuration from the previous post:

"logs": { "logs_collected": { "files": { "collect_list": [
  { "file_path": "/var/log/myapp/app.log", "log_group_name": "myapp/app", "log_stream_name": "{instance_id}", "timezone": "UTC" },
  { "file_path": "/var/log/nginx/access.log", "log_group_name": "myapp/nginx-access", "log_stream_name": "{instance_id}" }
]}}}

Prefer structured (JSON) logs with fields such as level, requestId, path, status and durationMs; Insights discovers JSON fields automatically. Never log passwords, tokens or full card numbers.

Retention and cost control

By default log groups never expire, and storage costs accumulate silently.

aws logs put-retention-policy --log-group-name myapp/app --retention-in-days 30
aws logs describe-log-groups --query 'logGroups[?retentionInDays==`null`].logGroupName' --output text

Set retention on every group (30 days for debug logs, a year or more for audit logs, subject to policy), export older logs to S3 with lifecycle rules, and avoid verbose debug logging in production.

Logs Insights queries

Choose the log group(s) and time range in the console, then:

# 1. Latest errors
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

# 2. Request count and p95 latency per path (JSON logs)
fields path, durationMs
| stats count() as requests, pct(durationMs, 95) as p95, avg(durationMs) as avg by path
| sort requests desc
| limit 20

# 3. Error rate per 5 minutes
filter status >= 500
| stats count() as errors by bin(5m)

# 4. Parse an unstructured nginx access line
parse @message '* - - [*] "* * *" * *' as ip, ts, method, url, proto, status, bytes
| filter status = 404
| stats count() as hits by url
| sort hits desc
| limit 10

Command	Purpose
fields	Choose columns
filter	Keep matching events (like, =, >, regular expressions)
parse	Extract fields from text with a glob or regex
stats ... by	Aggregate (count, sum, avg, min, max, pct)
sort, limit	Order and cut
bin(5m)	Group into time buckets

From the CLI (useful in runbooks):

QID=$(aws logs start-query --log-group-name myapp/app \
  --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
  --query-string 'filter @message like /ERROR/ | stats count() by bin(5m)' --query queryId --output text)
sleep 5; aws logs get-query-results --query-id $QID

Metric filters: turn logs into alarms

aws logs put-metric-filter --log-group-name myapp/app --filter-name errors \
  --filter-pattern '"ERROR"' \
  --metric-transformations metricName=AppErrors,metricNamespace=MyApp,metricValue=1,defaultValue=0

Then create a CloudWatch alarm on MyApp/AppErrors (see the alarms post). Metric filters only apply to events ingested after the filter is created.

Incident workflow

1. Identify the time window from the alarm.
2. Query errors per minute (stats count() by bin(1m)) to see the shape.
3. Narrow by requestId, user or path to find a sample failing request.
4. Query the neighbouring log groups (load balancer, database slow log) for the same window.
5. Save the useful queries (Insights supports saved queries) so the next on-call can reuse them.