CloudWatch Agent: Memory and Disk Metrics

By default EC2 reports CPU, network and status checks to CloudWatch, but it cannot see inside the operating system, so there is no memory or disk-space metric. The unified CloudWatch agent runs on the instance and publishes those, plus log files, as custom metrics.

State of AWS on this date. The unified agent (amazon-cloudwatch-agent) replaces the older monitoring scripts and the separate CloudWatch Logs agent. It runs on Linux and Windows. Custom metrics are billed per metric per month, so collect what you will alarm on.

How data gets to CloudWatch

flowchart TB
  CFG[Agent config JSON<br/>in SSM Parameter Store] --> AG
  subgraph EC2[EC2 instance with instance profile]
    AG[amazon-cloudwatch-agent] --> M1[mem and swap from /proc]
    AG --> M2[disk and diskio from filesystems]
    AG --> LG[tail log files]
  end
  ROLE[IAM role CloudWatchAgentServerPolicy] -.->|permissions| AG
  AG -->|PutMetricData| CWM[CloudWatch metrics<br/>namespace CWAgent]
  AG -->|PutLogEvents| CWL[CloudWatch Logs groups]
  CWM --> AL[Alarms on mem and disk above 80%]
  AL --> SNS[SNS topic ops-alerts]

1. Give the instance permission

Create an IAM role for EC2 with the AWS-managed policies CloudWatchAgentServerPolicy (publish metrics and logs) and, if you will pull config from Systems Manager, AmazonSSMManagedInstanceCore. Attach it as the instance profile; no keys are stored on the server.

aws iam create-role --role-name cw-agent-role --assume-role-policy-document file://ec2-trust.json
aws iam attach-role-policy --role-name cw-agent-role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam create-instance-profile --instance-profile-name cw-agent-profile
aws iam add-role-to-instance-profile --instance-profile-name cw-agent-profile --role-name cw-agent-role
aws ec2 associate-iam-instance-profile --instance-id i-0abc123 --iam-instance-profile Name=cw-agent-profile

2. Install and configure

sudo yum install -y amazon-cloudwatch-agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard     # optional guided config

A hand-written config, /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:

{
  "agent": { "metrics_collection_interval": 60, "run_as_user": "cwagent" },
  "metrics": {
    "append_dimensions": { "InstanceId": "${aws:InstanceId}", "AutoScalingGroupName": "${aws:AutoScalingGroupName}" },
    "metrics_collected": {
      "mem":  { "measurement": ["mem_used_percent"] },
      "swap": { "measurement": ["swap_used_percent"] },
      "disk": {
        "measurement": ["used_percent", "inodes_free"],
        "resources": ["/", "/data"],
        "ignore_file_system_types": ["tmpfs", "devtmpfs"]
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          { "file_path": "/var/log/messages", "log_group_name": "web/messages", "log_stream_name": "{instance_id}" }
        ]
      }
    }
  }
}

Start it and check:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
tail -n 30 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
aws cloudwatch list-metrics --namespace CWAgent --query 'Metrics[].MetricName' --output text | tr '\t' '\n' | sort -u

Metrics appear in the CWAgent namespace within a couple of minutes, with dimensions you chose (instance ID, device, filesystem path).

3. Keep one config for many servers

Store the JSON in SSM Parameter Store and have every instance fetch it:

aws ssm put-parameter --name AmazonCloudWatch-linux-web --type String --value file://amazon-cloudwatch-agent.json --overwrite
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux-web -s

Put the install and fetch commands in your AMI build or user data, and use the Systems Manager State Manager to keep every instance on the same config.

4. Alarms on what the agent reports

aws cloudwatch put-metric-alarm --alarm-name web01-mem-high \
  --namespace CWAgent --metric-name mem_used_percent \
  --dimensions Name=InstanceId,Value=i-0abc123 \
  --statistic Average --period 300 --evaluation-periods 3 --threshold 85 \
  --comparison-operator GreaterThanThreshold --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:ap-south-1:111111111111:ops-alerts

Dimensions must match exactly what the agent publishes, including device, fstype and path for disk metrics; list them with list-metrics and copy. breaching for missing data turns a dead agent or instance into an alert.

Troubleshooting

Symptom	Check
No metrics appear	Instance profile attached? agent.log shows AccessDenied? Outbound HTTPS to CloudWatch endpoints allowed?
Disk metrics missing	The resources path must be a mount point; use "*" for all
Alarm shows no data	Dimension mismatch, or the wrong region
High cost	Too many custom metrics: restrict resources, intervals and measurements

Remember the agent must also be installed and configured through your build process, or new instances will silently lack monitoring.