Persistent Volumes and Claims

A container’s filesystem disappears with the container. Databases, uploads and queues need storage that outlives pods. Kubernetes separates who needs storage (a PersistentVolumeClaim, PVC) from where it comes from (a PersistentVolume, PV, created by an administrator or, normally, on demand through a StorageClass).

Versions in use on this date. Kubernetes 1.22. In-tree cloud volume plugins are being migrated to CSI (Container Storage Interface) drivers, so on AWS the Amazon EBS CSI driver is the forward path. VolumeSnapshot has been stable since 1.20. ReadWriteOncePod access mode arrived as an alpha feature in 1.22.

Claim to disk

flowchart TB
  DEV[Developer writes PVC<br/>10Gi RWO class gp3] --> PVC[PersistentVolumeClaim]
  SC[StorageClass gp3<br/>provisioner ebs.csi.aws.com] --> PROV
  PVC -->|no matching PV, dynamic provisioning| PROV[CSI external provisioner]
  PROV -->|CreateVolume| CLOUD[(Cloud disk EBS vol-0abc)]
  PROV --> PV[PersistentVolume pvc-1234 bound to claim]
  PVC --- PV
  POD[Pod mounts claim] -->|scheduler waits for first consumer| NODE[Node in same AZ as disk]
  NODE -->|CSI node plugin attach and mount| CLOUD
  POD -. reads and writes .-> NODE

StorageClass and claim

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations: { storageclass.kubernetes.io/is-default-class: "true" }
provisioner: ebs.csi.aws.com
parameters: { type: gp3, fsType: ext4, encrypted: "true" }
reclaimPolicy: Delete                      # use Retain for data you must not lose by accident
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer    # create the disk in the AZ where the pod lands
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: pgdata, namespace: shop }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3
  resources: { requests: { storage: 20Gi } }

Use the claim in a pod (or, for databases, a StatefulSet with volumeClaimTemplates, which creates one claim per replica):

spec:
  containers:
    - name: postgres
      image: postgres:13
      volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data, subPath: pgdata }]
  volumes:
    - name: data
      persistentVolumeClaim: { claimName: pgdata }

kubectl -n shop get pvc,pv
kubectl -n shop describe pvc pgdata            # events show provisioning errors
kubectl get storageclass

Key concepts

Concept	Meaning
Access modes	`ReadWriteOnce` (one node at a time; EBS), `ReadOnlyMany`, `ReadWriteMany` (needs NFS, EFS, CephFS…)
Reclaim policy	`Delete` removes the disk when the PVC is deleted; `Retain` keeps it for manual recovery
Binding mode	`WaitForFirstConsumer` avoids creating a zonal disk in a zone where the pod cannot run
Expansion	Edit the PVC’s `requests.storage` upward; the driver grows the disk and the filesystem (online for most drivers); shrinking is not supported
CSI driver	A plugin that creates, attaches and mounts volumes for one storage system

Because an EBS volume lives in one Availability Zone and attaches to one node, a pod using it can only run in that zone. A node failure in that AZ stops the pod from moving elsewhere, which is why databases should replicate at the application level and have backups.

Grow, snapshot, restore

kubectl -n shop patch pvc pgdata -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
kubectl -n shop get pvc pgdata -w                       # watch FileSystemResizePending, then the new capacity

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: { name: pgdata-2021-09-06, namespace: shop }
spec:
  volumeSnapshotClassName: ebs-snapclass
  source: { persistentVolumeClaimName: pgdata }
---
# restore into a new claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: pgdata-restore, namespace: shop }
spec:
  storageClassName: gp3
  dataSource: { name: pgdata-2021-09-06, kind: VolumeSnapshot, apiGroup: snapshot.storage.k8s.io }
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 50Gi } }

Snapshots need the snapshot CRDs and controller installed plus a VolumeSnapshotClass. A crash-consistent disk snapshot of a busy database may need recovery; for guaranteed consistency, also take logical dumps (pg_dump) or use the database’s own backup tool. Velero backs up objects and volumes together to object storage.

Operating storage

Set resource quotas for storage per namespace: requests.storage and persistentvolumeclaims.
Alert on volume usage (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.8) from Prometheus.
Protect important data with reclaimPolicy: Retain, and deleting a StatefulSet does not delete its PVCs.
Encrypt volumes and use backups tested by actual restores.

Troubleshooting

Symptom	Cause and check
PVC `Pending`	No default StorageClass, wrong class name, driver not installed, or waiting for first consumer; read `describe pvc` events
Pod `ContainerCreating`, “Multi-Attach error”	The RWO volume is still attached to another node; wait for detach or fix the old pod
Pod stuck `Pending` with “volume node affinity conflict”	Disk exists in one AZ, pod scheduled in another; use `WaitForFirstConsumer`
Resize stuck	Driver or class lacks expansion; the pod may need a restart for filesystem resize
Data lost on delete	`reclaimPolicy: Delete`; restore from snapshot or backup