MongoDB 7.0: Replica Set, Failover and Backups You Can Restore

One MongoDB server is a single point of failure. A replica set keeps copies of the data on several members and elects a new primary automatically. This post builds a three-member set on MongoDB 7.0 (8.0 is still in release-candidate stage on this date), then covers backups, the part many teams get wrong.

Architecture

flowchart TB
  APP[Application driver<br/>replicaSet rs0 w majority] -->|writes and reads| P[Primary db1]
  P -->|oplog replication| S[Secondary db2<br/>priority 1 voting]
  P -->|oplog replication| H[Hidden member db3<br/>priority 0 voting<br/>used for backups]
  S <-->|heartbeats and elections| H
  S <--> P
  H -->|mongodump or snapshots| BK[Backup storage]
  H -->|oplog slices| PBM[Percona Backup for MongoDB<br/>optional]
  PBM --> BK

• Use an odd number of voting members (three) so a majority exists after one failure. Avoid arbiters; they vote but hold no data, which weakens w: majority durability.
• A hidden member has priority: 0 and hidden: true. Applications never read from it, so backups and reports do not disturb production traffic.
• Place members in different failure domains (racks or availability zones).

Configure every member

Same settings on all three servers, plus a shared keyfile for internal authentication (keyfile auth also enables authorization):

openssl rand -base64 756 | sudo tee /etc/mongodb-keyfile
sudo chown mongodb:mongodb /etc/mongodb-keyfile && sudo chmod 400 /etc/mongodb-keyfile

# /etc/mongod.conf (on db1, db2, db3)
net:
  bindIp: 127.0.0.1,10.0.1.21        # each host's own private IP
  tls: { mode: requireTLS, certificateKeyFile: /etc/ssl/mongodb/server.pem, CAFile: /etc/ssl/mongodb/ca.pem }
security:
  keyFile: /etc/mongodb-keyfile
  authorization: enabled
replication:
  replSetName: rs0
  oplogSizeMB: 4096

Use DNS names that resolve on every member, not IP addresses that may change.

Initiate the set

Connect to db1 (create the admin user first, as in the previous post), then:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "db1.example.com:27017", priority: 2 },
    { _id: 1, host: "db2.example.com:27017", priority: 1 },
    { _id: 2, host: "db3.example.com:27017", priority: 0, hidden: true }
  ]
})
rs.status()
rs.conf()

Connection string for applications:

mongodb://shop_app:***@db1.example.com,db2.example.com/shop?replicaSet=rs0&tls=true&w=majority&retryWrites=true

w=majority makes a write acknowledged only after most members have it, so it survives a failover. Rehearse a failover with rs.stepDown(60) on the primary and watch the driver reconnect without errors.

Backup strategy

Method	Good for	Limits
mongodump / mongorestore	Small and medium data, logical copies, selective restores	Slow on large data; not a consistent snapshot of a busy set unless --oplog is used
Filesystem or volume snapshot of the hidden member	Large data, fast restore	Needs journal and data on one volume, or fsyncLock
Percona Backup for MongoDB (open source)	Point-in-time recovery from oplog slices	Extra agents and storage to run

Dump from the hidden member, with an oplog for consistency:

mongodump --host db3.example.com --tls --tlsCAFile /etc/ssl/mongodb/ca.pem \
  -u backup -p "$BACKUP_PW" --authenticationDatabase admin \
  --oplog --gzip --out /backup/$(date +%F)

The backup user needs the built-in backup role. Copy dumps off the host, keep a retention schedule (for example 7 daily and 4 weekly), and encrypt backups at rest.

Restore test (monthly):

mongorestore --host restore-test --tls --tlsCAFile ca.pem -u admin -p --authenticationDatabase admin \
  --oplogReplay --gzip /backup/2024-08-19

Verify document counts and run an application smoke test. A backup that has never been restored is not yet a backup.

Monitoring and failure modes

• rs.printSecondaryReplicationInfo() shows lag; alert if lag stays above 10 seconds.
• Size the oplog for the longest outage or backup window you need to survive; a secondary that falls off the oplog needs a full resync.
• No primary elected: fewer than a majority of voting members are reachable; fix the network or add a member.
• Rollbacks: writes acknowledged with w:1 on a primary that then fails can be rolled back; use w: majority for data that matters.
• Upgrading: upgrade secondaries first, step down the primary, then upgrade it; set featureCompatibilityVersion last.