CloudWatch Metrics and Alarms

CloudWatch is AWS’s built-in monitoring service. It stores metrics (numbers over time), evaluates alarms against them and can notify people or trigger actions. This post sets up alerting for an EC2 instance the way you would for production.

State of AWS on this date. EC2 sends basic metrics every 5 minutes for free; detailed monitoring (1-minute data) costs extra per instance. Composite alarms, which combine other alarms to cut noise, were added in March 2020. Memory and disk usage are not collected by default (the next post adds them with the CloudWatch agent).

From metric to action

flowchart TB
  EC2[EC2 instance<br/>CPUUtilization StatusCheckFailed] -->|every 5 min| MET[CloudWatch metric<br/>namespace AWS/EC2]
  MET --> AL{Alarm evaluates<br/>threshold, periods, statistic}
  AL -->|OK to ALARM| SNS[SNS topic ops-alerts]
  SNS --> EM[Email]
  SNS --> CH[Chat webhook via Lambda]
  AL -->|ALARM| ACT[Alarm action<br/>EC2 recover or reboot]
  AL -->|INSUFFICIENT_DATA| ID[No data - treat per setting]
  MET --> DB[Dashboard graphs]

What EC2 gives you by default

Metric	Meaning	Alarm worth setting
CPUUtilization	% of allocated CPU in use	Above 80% for 15 minutes
StatusCheckFailed_System	AWS host problem	Any failure: recover the instance
StatusCheckFailed_Instance	OS or network problem inside the VM	Failure for 2 periods: notify or reboot
NetworkIn / NetworkOut	Bytes per period	Unusual spikes
DiskReadOps / DiskWriteOps	Instance store disks only	Rarely useful for EBS
CPUCreditBalance (t2/t3)	Burst credits left	Below 20: you are about to be throttled

For EBS volumes, use VolumeQueueLength and BurstBalance (gp2 volumes). Application load balancers, RDS, SQS and other services publish their own metrics.

1. Notification topic

aws sns create-topic --name ops-alerts
aws sns subscribe --topic-arn arn:aws:sns:ap-south-1:111111111111:ops-alerts \
    --protocol email --notification-endpoint ops@example.com     # confirm the email!

An unconfirmed subscription receives nothing, so check that the confirmation link was clicked, and send a test message with aws sns publish.

2. CPU alarm

aws cloudwatch put-metric-alarm \
  --alarm-name web01-cpu-high \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123def456 \
  --statistic Average --period 300 --evaluation-periods 3 --datapoints-to-alarm 3 \
  --threshold 80 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:ap-south-1:111111111111:ops-alerts \
  --ok-actions arn:aws:sns:ap-south-1:111111111111:ops-alerts

Read it as: “the average CPU over 5-minute periods is above 80% for 3 consecutive periods”. datapoints-to-alarm allows “M out of N” logic. Always set --ok-actions so people know when the problem clears.

3. Auto-recover on host failure

aws cloudwatch put-metric-alarm \
  --alarm-name web01-system-check \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_System \
  --dimensions Name=InstanceId,Value=i-0abc123def456 \
  --statistic Maximum --period 60 --evaluation-periods 2 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:automate:ap-south-1:ec2:recover arn:aws:sns:ap-south-1:111111111111:ops-alerts

The recover action restarts the instance on healthy hardware with the same ID, IP and volumes. It works for EBS-backed instances of supported types, not for instance-store ones.

Alarm states and missing data

• OK, ALARM, INSUFFICIENT_DATA.
• treat-missing-data: missing (default; state held), notBreaching (good for sparse metrics), breaching (use for heartbeat metrics where silence means failure), ignore.
• A stopped instance stops publishing metrics, so a CPU alarm moves to INSUFFICIENT_DATA, not ALARM. If you need “instance is down”, alarm on StatusCheckFailed or on a load-balancer health metric.

Composite alarms and metric math

Composite alarms reduce noise by combining conditions:

aws cloudwatch put-composite-alarm --alarm-name web01-sick \
  --alarm-rule 'ALARM("web01-cpu-high") AND ALARM("web01-5xx-high")' \
  --alarm-actions arn:aws:sns:ap-south-1:111111111111:ops-alerts

Metric math (--metrics with expressions) lets you alarm on rates and percentages, such as errors divided by requests.

Alarm design rules

1. Alert on symptoms users feel (errors, latency, availability) and on capacity limits; fewer, meaningful alarms beat hundreds ignored.
2. Every alarm needs an owner and a runbook line saying what to do.
3. Pick periods that match the data: a 5-minute metric cannot detect a 1-minute spike.
4. Test: aws cloudwatch set-alarm-state --alarm-name web01-cpu-high --state-value ALARM --state-reason test and confirm the email arrives.
5. Create alarms as code (CloudFormation or Terraform) so they exist for every new instance.