Backups with rsync and tar

Disks fail, people delete the wrong directory and software upgrades go wrong. A backup you have never restored is not yet a backup. This post builds a small, dependable routine with tools already on every server, and a habit of testing the restore.

Versions in use on this date. rsync 3.1.2 and GNU tar 1.29 on Ubuntu 18.04.3 LTS; rsync 3.1.2 and tar 1.26 on CentOS 7.7. The examples run on both.

Design: the 3-2-1 idea

Keep 3 copies of the data, on 2 different kinds of storage, with 1 copy off-site. For a single server that means: the live data, a local or attached backup, and a copy on another host or object storage.

flowchart TB
  SRV[Production server<br/>/etc /var/www /home] --> DB[Database dump first<br/>consistent copy]
  DB --> TAR[tar archive with date in name]
  SRV --> TAR
  TAR --> LOC[Local backup disk /backup]
  LOC -->|rsync over SSH| REM[Remote backup host<br/>different location]
  REM --> RET[Retention - 7 daily 4 weekly 6 monthly]
  REM --> TEST[Monthly restore test on a spare VM]
  CRON[cron nightly with flock] -.-> TAR

What to back up

• Configuration: /etc, crontabs, systemd units you created, TLS certificates.
• Application data: web roots, uploads, /var/lib of your services.
• Databases: dump them, instead of copying live data files. For MySQL: mysqldump --single-transaction --all-databases | gzip > db-$(date +%F).sql.gz. For PostgreSQL: pg_dumpall or pg_dump -Fc.
• A list of installed packages: dpkg --get-selections > packages.txt (or rpm -qa).

Do not back up what can be rebuilt (caches, /proc, package files). Keep the backup under a different account and permissions, so malware or a mistake on the server cannot also delete it.

tar: one archive per run

sudo tar --create --gzip --file /backup/etc-$(date +%F).tar.gz \
    --exclude='*.tmp' -C / etc var/www
tar --list --gzip --file /backup/etc-2019-12-16.tar.gz | head
tar --extract --gzip --file /backup/etc-2019-12-16.tar.gz -C /tmp/restore etc/nginx/nginx.conf

-C / stores relative paths, so extraction goes where you choose instead of overwriting live files. tar preserves permissions and ownership when run as root (--same-owner, the default for root), which matters for restores.

rsync: efficient copies

rsync -aHAX --delete --numeric-ids /var/www/ /backup/www/            # local mirror
rsync -az --delete -e "ssh -i ~/.ssh/backup_key" /backup/ backup@host2:/srv/backups/web01/
rsync -avn --delete /var/www/ /backup/www/                            # -n dry run: always preview

Option	Why
-a	Archive: recursive, keeps permissions, times, symlinks
-H -A -X	Hard links, ACLs, extended attributes
--delete	Remove files at the destination that no longer exist at the source
--numeric-ids	Preserve raw UID/GID, avoids name mismatches
-n	Dry run
--bwlimit=5000	Limit bandwidth (KB/s)

Mind the trailing slash: src/ copies the contents, src copies the directory itself. --delete mirrors deletions too, so a mirror is not a history; keep dated archives or snapshots as well.

Snapshots with hard links

--link-dest creates daily directories that look like full copies but store only changed files:

today=/backup/web01/$(date +%F); last=/backup/web01/latest
rsync -a --delete --link-dest="$last" /var/www/ "$today/"
ln -sfn "$today" "$last"

Script and schedule

#!/usr/bin/env bash
set -euo pipefail
d=$(date +%F); dest=/backup/$(hostname -s)
mkdir -p "$dest"
mysqldump --single-transaction --all-databases | gzip > "$dest/db-$d.sql.gz"
tar -czf "$dest/files-$d.tar.gz" -C / etc var/www
tar -tzf "$dest/files-$d.tar.gz" >/dev/null                 # verify readable
find "$dest" -type f -mtime +7 -delete                      # local retention
rsync -az --partial -e "ssh -i /root/.ssh/backup_key" "$dest/" backup@host2:/srv/backups/$(hostname -s)/
echo "backup ok $d"

0 2 * * * flock -n /var/lock/backup.lock /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Restrict the remote key in authorized_keys (from=, no-pty) and protect the backup host.

Restore tests: the part that matters

1. Monthly, restore the latest backup onto a spare VM, not onto production.
2. Start the service and check real data (record counts, a login, a page).
3. Time it, and write down the steps and the time taken: that is your recovery time.
4. Test restoring a single file and a full server, since they use different steps.
5. Alert when a backup is missing or suspiciously small: find /backup -name 'files-*' -mtime -1 -size +1M | grep -q . || echo "BACKUP MISSING".

Encrypt off-site copies (gpg --symmetric or an encrypted volume) and store the key separately, since a backup that anyone can read is also a data leak.