MongoDB handles an unclean shutdown well because of its journal, so most crashes recover on their own. When a pod stays in CrashLoopBackOff after a node reboot, a full disk or an out-of-memory kill, the log tells you which of a handful of causes you have. This post covers the recovery order, from least to most destructive.
Versions in use on this date. MongoDB 8.0 (released October 2024) on MicroK8s 1.32 with the hostpath-storage add-on. MongoDB 7.0 is still supported, and the steps are the same. The setup is a three-member replica set in a StatefulSet, each member with its own volume.
Decide by the symptoms
flowchart TB
A[mongo-1 CrashLoopBackOff] --> B[kubectl logs --previous]
B --> C{Error}
C -->|Permission denied or keyfile| K[Fix owner and mode]
C -->|No space left| D[Free disk or grow volume]
C -->|WT_PANIC or corrupt file| R[Resync the member]
C -->|Starts but not in set| S[Fix replica set config]
R --> Q{Other members healthy}
Q -->|Yes| RS[Wipe data and initial sync]
Q -->|No| RP[mongod repair on a copy]
The golden rule: if the other two members are healthy, a broken member is cheap, because it can copy everything from them. If it is a single standalone instance, treat the volume as precious and work on a copy.
Step 1: read the log
microk8s kubectl get pods -n data -l app=mongo
microk8s kubectl logs mongo-1 -n data --previous --tail=80
microk8s kubectl describe pod mongo-1 -n data | grep -E 'Reason|Exit Code|OOM'| Message | Meaning |
|---|---|
Permissions on /etc/mongo/keyfile are too open |
Keyfile mode must be 400 or 600, owned by the mongo user |
Failed to start up WiredTiger or WT_PANIC |
Data files damaged |
Detected unclean shutdown - mongod.lock is not empty |
Normal after a crash; the journal is replayed |
Data directory ... has been created by a different version |
Version or feature compatibility mismatch |
No space left on device |
Disk full; MongoDB cannot even write its journal |
Step 2: check the replica set from a healthy member
microk8s kubectl exec mongo-0 -n data -- mongosh -u admin -p "$MONGO_PW" --authenticationDatabase admin \
--eval 'rs.status().members.map(m => ({n:m.name, state:m.stateStr, lag:m.optimeDate}))'A healthy primary and one other secondary mean the set still has a majority (2 of 3) and keeps accepting writes. Do not remove the broken member from the config yet.
Step 3: fix the small things first
Keyfile and ownership. The official image runs as user 999. Set securityContext: { runAsUser: 999, fsGroup: 999 }, and mount the keyfile secret with defaultMode: 0400. On the node, a restored folder may need sudo chown -R 999:999 <volume>.
Disk full.
df -h /var/snap/microk8s/common
sudo du -sh /var/snap/microk8s/common/default-storage/*Free node space (logs, microk8s ctr images prune), then start the pod. If the volume is the problem, expand it and the node disk; hostpath volumes share the node’s filesystem, so the limit is the node disk.
Out of memory. WiredTiger takes about half of (RAM minus 1 GB) for its cache by default, but inside a container it must be told the limit. Set --wiredTigerCacheSizeGB 1 for a pod with a 2 GiB limit, or you will see repeated OOMKilled.
Step 4: resync a damaged member (preferred)
With two good members, discard the damaged one’s data and let MongoDB copy it back (an initial sync):
microk8s kubectl scale statefulset mongo -n data --replicas=2 # stops the highest ordinalIf the broken member is not the last ordinal, instead run a temporary pod that mounts its PVC and empties it, or move the folder aside on the node:
VOL=$(sudo ls -d /var/snap/microk8s/common/default-storage/*mongo-1*)
sudo mv "$VOL" "${VOL}.broken-$(date +%F)"
sudo mkdir "$VOL" && sudo chown 999:999 "$VOL"
microk8s kubectl delete pod mongo-1 -n dataThe member starts empty, joins as STARTUP2 and copies the data from the primary. Watch it reach SECONDARY:
watch -n5 'microk8s kubectl exec mongo-0 -n data -- mongosh -u admin -p "$MONGO_PW" --authenticationDatabase admin --quiet --eval "rs.status().members.forEach(m=>print(m.name,m.stateStr))"'Initial sync of a large database takes a long time and uses network and disk on the primary, so do it outside busy hours. The oplog must be large enough to cover the sync time (rs.printReplicationInfo()), otherwise the member falls off the end and starts again.
Step 5: repair a lone instance (last resort)
Only if no healthy copy exists. Work on a copy:
sudo cp -a "$VOL" /root/mongo-backup-$(date +%F-%H%M)
microk8s kubectl run mongo-repair -n data --rm -it --restart=Never --image=mongo:8.0 \
--overrides='{"spec":{"securityContext":{"runAsUser":999},"containers":[{"name":"mongo-repair","image":"mongo:8.0","command":["mongod","--dbpath","/data/db","--repair"],"volumeMounts":[{"name":"d","mountPath":"/data/db"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"data-mongo-0"}}]}}'--repair salvages what it can and drops what it cannot read, so collections can lose documents. After it finishes, start normally, run db.collection.validate({full:true}) on the important collections, and dump everything at once with mongodump. Then rebuild the instance from that dump.
After recovery
Confirm one primary and two secondaries, zero lag, and the application writing again. Then fix the cause: memory limits and cache size, a PodDisruptionBudget, a disk alert, and a tested backup (mongodump --oplog to another machine, or snapshots). A replica set protects against a member failing, but it does not protect against deletion or corruption copied to every member.