A nightly systemd timer gzips a pg_dump
and snapshots the signing keys under /etc/qnt. Wire
the optional R2 mirror to get the dumps off-host, and
keep the restore drill handy so a bad migration is a 5-minute fix,
not an outage.
Two things, both captured by qnt-backup.sh: the
Postgres database, and the secret keys under /etc/qnt.
The database is the obvious one — the keys are the one people
forget, and the one that hurts most to lose.
A gzip'd pg_dump. The script defaults to
PG_USER=qnt, PG_DB=qnt,
PG_HOST=127.0.0.1, and reads a
DATABASE_URL from
/etc/qnt/qnt-server.env to pick up the DB password
if one is set.
Dumps land in /var/backups/qnt
(BACKUP_DIR=/var/backups/qnt in the script).
The script also snapshots /etc/qnt/*.key and
server.crt:
Without these, every customer's capability token, every active session, and OAuth login break. The database alone is not a recoverable backup.
Three files ship in the repo:
deploy/qnt-backup.sh,
deploy/qnt-backup.service, and
deploy/qnt-backup.timer. Copy them into place and
enable the timer.
cp deploy/qnt-backup.sh /usr/local/bin/qnt-backup.sh
chmod +x /usr/local/bin/qnt-backup.sh
cp deploy/qnt-backup.service /etc/systemd/system/
cp deploy/qnt-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now qnt-backup.timerOnCalendar=*-*-* 03:17:00 — 03:17 UTC daily,
offset off the top of the hour to dodge the 03:00 cron
storm.Persistent=true — if the VM was off at 03:17,
the run fires on next boot rather than being skipped.RandomizedDelaySec=5min — jitter so the run
doesn't land at exactly the same instant every night.To take a backup on demand — before a migration, say — run the script directly:
sudo /usr/local/bin/qnt-backup.shRead this before you decide you're done. A local nightly dump is a real safety net, but it has one sharp limit.
This only protects against logical loss
(DROP TABLE, a bad migration). It does
not protect against the disk failing — backups
live on the same VM. Off-host mirroring is a
separate hardening pass.
In other words: if the VM's disk dies, the dumps die with it. The next section wires the optional off-host mirror to close that gap.
Optional and opt-in. deploy/pg-backup-r2-push.sh
mirrors the dumps to a Cloudflare R2 bucket so a dead disk doesn't
take the backups with it.
The push script soft-skips when rclone or its
config is absent, so you can wire it into your flow before R2 is
actually set up — it just does nothing until the pieces are in
place.
rclone.Once rclone and the remote are configured, the off-host copy rides along with the nightly run — no extra schedule to manage.
Practice this before you need it. The sequence below restores a
gzip'd SQL dump from /var/backups/qnt — and restores
the keys first if you lost the disk.
If this is a disk-loss recovery, restore
/etc/qnt/*.key from the key snapshots
before starting qnt-server. Skip this and every
session and capability token breaks even after the DB is back.
On a logical-loss restore (the keys never went anywhere) you can jump straight to the DB steps below.
Stop qnt-server, pick the latest dump from
/var/backups/qnt, then drop and recreate the
database:
systemctl stop qnt-server
sudo -u postgres dropdb qnt
sudo -u postgres createdb qnt
Pipe the gzip'd dump straight into psql (swap
<dump> for the file you picked):
gunzip -c /var/backups/qnt/<dump>.sql.gz | sudo -u postgres psql -d qntsudo -u postgres psql -d qnt -c "SELECT count(*) FROM tunnels"
systemctl start qnt-server
systemctl is-active qnt-server
A row count from tunnels confirms the data is back;
is-active returning active confirms
qnt-server came up clean against the restored database.
Backups armed and a drill in hand. Round out the operational guides.