mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

Archive · AWS

Secrets Manager and KMS

Store credentials in AWS Secrets Manager, read them with least-privilege roles, rotate them automatically and control the KMS keys behind them, with notes on Parameter Store and Kubernetes.

5 min read

  • AWS Secrets Manager
  • AWS KMS
  • AWS CLI 2.9
  • SSM Parameter Store
  • External Secrets Operator

Do not put passwords in code, container images or environment files in Git. Anything committed is copied to every clone, CI cache and backup, and removing it later does not make it secret again. Instead, keep secrets in a service built for them, give each workload permission to read only its own, and rotate them on a schedule.

How the pieces connect

flowchart TB
  ADM[Admin or pipeline] -->|create secret| SM[(Secrets Manager<br/>prod/shop/db)]
  SM -->|encrypted with| KMS[KMS customer managed key<br/>key policy and rotation]
  APP[Application with IAM role<br/>ECS task role, Lambda role, EC2 profile] -->|GetSecretValue on that ARN| SM
  SM -->|plaintext over TLS, cached by app| APP
  APP --> DB[(RDS database)]
  ROT[Rotation Lambda on schedule] -->|set new password| DB
  ROT -->|put new version AWSPENDING then AWSCURRENT| SM
  CT[CloudTrail] -.->|logs every Decrypt and GetSecretValue| SM
  CT -.-> KMS

Create and read a secret

aws secretsmanager create-secret --name prod/shop/db --description "Shop app database login" \
  --kms-key-id alias/shop-secrets \
  --secret-string "$(jq -n --arg p "$(openssl rand -base64 24)" '{username:"app",password:$p}')" \
  --region ap-south-1

aws secretsmanager get-secret-value --secret-id prod/shop/db --query SecretString --output text
aws secretsmanager list-secret-version-ids --secret-id prod/shop/db

Generating the value in the command means it is never typed into a terminal history or chat. Secrets keep versions with staging labels: AWSCURRENT is what applications read, AWSPREVIOUS is the last one, and AWSPENDING is used during rotation, which makes rollback easy.

Least-privilege access

The workload’s role gets one statement for its own secret, plus KMS permission if you use a customer managed key:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:ap-south-1:111122223333:secret:prod/shop/db-*" },
    { "Effect": "Allow", "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:ap-south-1:111122223333:key/1111aaaa-22bb-33cc-44dd-555555eeeeee",
      "Condition": { "StringEquals": { "kms:ViaService": "secretsmanager.ap-south-1.amazonaws.com" } } }
  ]
}

Reading in code (Python), done once at start-up and refreshed periodically:

import json, boto3
client = boto3.client("secretsmanager", region_name="ap-south-1")
db = json.loads(client.get_secret_value(SecretId="prod/shop/db")["SecretString"])

Use the role from the platform (task role, Lambda execution role, instance profile), never a stored access key. Cache the value (AWS provides caching libraries) to avoid an API call per request and rate limits, and re-read it on authentication failure so rotation does not cause an outage.

On ECS you can also inject a secret into an environment variable by referencing its ARN in the task definition; the value is resolved when the task starts, so a rotated secret needs a task restart.

Rotation

Secrets Manager can rotate credentials on a schedule using a Lambda function. For Amazon RDS it provides templates, so no one keeps a long-lived password:

aws secretsmanager rotate-secret --secret-id prod/shop/db \
  --rotation-lambda-arn arn:aws:lambda:ap-south-1:111122223333:function:SecretsManagerRDSPostgreSQLRotation \
  --rotation-rules AutomaticallyAfterDays=30

Rotation steps are createSecret, setSecret, testSecret and finishSecret. The Lambda needs network access to the database (in the VPC) and to the Secrets Manager endpoint. For zero downtime, use the alternating users strategy for busy databases, so the old credential still works while clients refresh.

Secrets Manager or Parameter Store?

Secrets Manager SSM Parameter Store SecureString
Cost About 0.40 USD per secret per month plus API calls Standard parameters have no charge
Built-in rotation Yes No (build your own)
Cross-region replication Yes No
Size limit 64 KB 4 KB standard, 8 KB advanced
Best for Database credentials, API keys that rotate Configuration values and simple secrets

Both use KMS and IAM, and both are logged in CloudTrail. Use Secrets Manager for credentials that must rotate, and Parameter Store for configuration and cheap, rarely changed secrets. Prices change; check the pricing page.

KMS in a few lines

  • Most AWS services encrypt with AWS managed keys (aliases like aws/secretsmanager) at no charge, and you cannot edit their policy.
  • A customer managed key (CMK) gives you your own key policy, grants, CloudTrail audit of every use, and optional automatic rotation every year of the key material. It costs about 1 USD per key per month plus requests.
  • The key policy is the primary access control: it must allow the account’s IAM to delegate (the default statement), and you add specific roles for use and for administration, kept separate.
  • Envelope encryption: KMS encrypts a small data key, which encrypts the data; the data itself never goes to KMS.
  • Deleting a key is scheduled with a 7 to 30 day wait because data encrypted with it becomes unrecoverable; alarm on ScheduleKeyDeletion events.
  • Multi-Region keys let you decrypt in a DR region without re-encrypting data.
aws kms create-key --description "Shop secrets" --key-usage ENCRYPT_DECRYPT
aws kms create-alias --alias-name alias/shop-secrets --target-key-id <key-id>
aws kms enable-key-rotation --key-id <key-id>

Kubernetes

On EKS or any cluster, avoid committing Kubernetes Secret manifests (they are only base64). Use the External Secrets Operator to sync from Secrets Manager into native secrets, or the Secrets Store CSI driver to mount them as files, with access through IAM roles for service accounts (IRSA).

Good habits

  1. One secret per application and environment, with a clear naming scheme (env/app/purpose) and tags for owner.
  2. Alarm on unusual GetSecretValue volume or access from unexpected principals, using CloudTrail and GuardDuty.
  3. Run a secret scanner (gitleaks, GitHub secret scanning) in CI. If a secret leaks, rotate first, then investigate.
  4. Never log secret values, and mask them in CI output.
  5. Include secrets and keys in your restore drill: backups are useless if the key to decrypt them is gone.