ConfigMaps and Secrets

An image should run unchanged in dev, staging and production; only its configuration differs. Kubernetes holds that configuration in ConfigMaps (ordinary settings) and Secrets (sensitive values), and injects it into pods as environment variables or files.

Versions in use on this date. Kubernetes 1.21 (April 2021), where immutable ConfigMaps and Secrets reached general availability. An immutable object cannot be changed (only deleted and recreated), which prevents accidental edits and reduces load on the API server for large clusters.

How configuration reaches a container

flowchart TB
  GIT[Git repo manifests<br/>non-secret config] --> CM[ConfigMap app-config]
  EXT[Secret store<br/>Vault AWS Secrets Manager] -->|operator or CSI driver| SEC[Secret db-credentials]
  CM -->|envFrom or valueFrom| ENV[Environment variables]
  CM -->|volume mount| FILES[Files in /etc/app]
  SEC -->|valueFrom secretKeyRef| ENV
  SEC -->|volume mount tmpfs| SF[Files in /etc/secrets]
  ENV --> POD[Pod container]
  FILES --> POD
  SF --> POD
  ETCD[(etcd)] -.->|Secrets stored here<br/>encrypt at rest| SEC

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v3
  namespace: shop
immutable: true
data:
  LOG_LEVEL: info
  FEATURE_X: "true"
  app.properties: |
    cache.ttl=300
    upstream=https://api.internal

kubectl create configmap app-config --from-file=app.properties --from-literal=LOG_LEVEL=info --dry-run=client -o yaml

Use it in a pod:

containers:
  - name: web
    image: registry.example.com/shop/web:1.4.2
    envFrom:
      - configMapRef: { name: app-config-v3 }
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef: { name: db-credentials, key: password }
    volumeMounts:
      - { name: cfg, mountPath: /etc/app, readOnly: true }
volumes:
  - name: cfg
    configMap: { name: app-config-v3 }

Env vars or files?

	Environment variable	Mounted file
Updates when the object changes	No, only at pod start	Yes, within about a minute (not for subPath mounts)
Fits	Simple values, 12-factor apps	Config files, certificates, larger content
Leak risk	Visible in kubectl describe, /proc, crash dumps and child processes	Safer permissions with defaultMode: 0400

Applications that only read config at startup need a restart anyway. A common pattern is immutable, versioned ConfigMaps (app-config-v3): create the new one, point the Deployment at it, and the rollout restarts pods cleanly with a clear rollback.

Many teams add a hash of the config as a pod annotation (Helm: checksum/config) to force a rollout when configuration changes.

Secrets

apiVersion: v1
kind: Secret
metadata: { name: db-credentials, namespace: shop }
type: Opaque
stringData:                  # plain text here, stored base64-encoded
  username: shop_app
  password: change-me

kubectl -n shop create secret generic db-credentials --from-literal=username=shop_app --from-file=password=./password.txt
kubectl -n shop create secret docker-registry regcred --docker-server=registry.example.com --docker-username=ci --docker-password="$TOKEN"
kubectl -n shop create secret tls web-tls --cert=tls.crt --key=tls.key

Be honest about what a Secret is. The values are only base64-encoded, not encrypted; anyone allowed to get the Secret (or exec into a pod that mounts it) can read them. Protect them in layers:

1. RBAC: give very few people and service accounts get/list on Secrets, and never use cluster-wide wildcards.
2. Encryption at rest in etcd, using an EncryptionConfiguration on self-managed clusters, or envelope encryption with a KMS key on EKS, GKE and AKS.
3. Keep real secrets out of Git. Use Sealed Secrets, SOPS or an external store, with the External Secrets Operator or the Secrets Store CSI driver syncing values into the cluster.
4. Mount as files with tight permissions, and avoid logging environment variables.
5. Rotate credentials and restart consumers; consider short-lived credentials (database users issued by Vault, cloud IAM roles) over long-lived passwords.
6. Never bake secrets into images or pass them as command-line arguments.

Limits and behaviour

• Each object may be at most 1 MiB.
• ConfigMaps and Secrets are namespaced; a pod can only use those in its own namespace.
• If a referenced object or key is missing, the pod stays in CreateContainerConfigError unless the reference is optional: true.
• Immutable objects cannot be edited: kubectl apply fails with field is immutable; create a new name instead.

Troubleshooting

kubectl -n shop get cm,secret
kubectl -n shop describe pod web-xxx | sed -n '/Events/,$p'
kubectl -n shop exec deploy/web -- env | sort | grep -v PASSWORD
kubectl -n shop get secret db-credentials -o jsonpath='{.data.password}' | base64 -d     # restricted by RBAC; use carefully
kubectl auth can-i get secrets -n shop --as=system:serviceaccount:shop:web

Check spelling of keys (key not found), namespaces, and whether a rollout actually restarted the pods after a change.