Disaster Recovery Bootstrap
The question was simple: if server2 dies tomorrow — hardware failure, provider goes down, accidental destruction — how fast can I rebuild it from zero? The answer before this work was "hours of manual setup, hoping I remember everything." The answer now is one script and about 20 minutes of automated restoration.
1. The problem
server2 runs the entire operational stack: Docker containers for a book library, document management, and monitoring. A Caddy reverse proxy with automatic Let's Encrypt certificates. 40+ custom shell scripts. Cron jobs for backups, reports, and maintenance. OpenClaw for automation. Git repos for journal and financial ledger. Vim config with custom key mappings and beancount shortcuts. All of it backed up nightly to server1, but no automated way to put it all back together.
2. The backup system
A daily cron at 04:10 UTC runs rsync from server2 to server1, syncing the entire home directory to /mnt/vps-backup/. Excludes are minimal — just package caches and virtualenvs that can be regenerated. A separate cron syncs the book library at 04:00. Total backup size is around 1.7 GB. This means server1 always has a <24 hour old copy of everything on server2.
3. What the bootstrap script does
The script assumes a fresh Debian VPS with root access and an SSH key that can reach server1. It runs in sequence:
4. System packages
Installs everything server2 needs: Docker and docker-compose, Python 3 with pip, Node.js (for OpenClaw), git, rsync, vim, fail2ban, ufw, and build essentials. Python packages include beancount, whisper, and their dependencies. The package list was generated by diffing a fresh Debian install against server2's current state.
5. Pull from backup
rsync pulls the entire /mnt/vps-backup/ from server1 back to the new machine's home directory. This restores all scripts, configs, dotfiles, journal, financial ledger, memory files, and application data in one shot. The SSH key to server1 is the only prerequisite — everything else flows from the backup.
6. Restore Docker services
The docker-compose files are part of the backup, so restoration is: navigate to each service directory and run docker-compose up -d. The book library needs its MariaDB data directory (included in the backup), the document manager needs its uploaded files (included), and the monitoring tool needs its SQLite database (included). Caddy config is also in the backup — it picks up the reverse proxy rules automatically.
7. Restore cron jobs
A crontab.backup file is maintained in the home directory and synced to server1 daily. The script loads it with crontab crontab.backup. This restores all 14 scheduled jobs — backup syncs, daily digests, school reports, financial reports, memory maintenance, and journal sync.
8. Fix git remotes
The journal and finance repos have origin URLs pointing to GitHub. After restoration, git pull/push works immediately because the SSH keys (for GitHub) are part of the home directory backup. The script verifies each repo can reach its remote.
9. Install OpenClaw
OpenClaw is installed via npm. The config, session history, cron state, and memory files are all in the backup. After installation, the agent resumes with full context — it knows the infrastructure, the user's preferences, and all scheduled tasks.
10. Validation
A separate validation script runs 44 checks across every component: Docker services responding on expected ports, cron jobs loaded, SSH key access to server1 working, git repos in clean state, all 40 bin scripts present and executable, Caddy serving HTTPS, DNS resolving correctly, fail2ban running, backup cron scheduled. Each check prints pass/fail with details.
11. Pre-restore checklist
A markdown checklist documents what to do before running the script: provision a new VPS at the same provider, ensure the SSH key to server1 is available (it's also backed up to an encrypted local copy), update DNS if the IP changes, and verify server1's backup is current.
12. Current system snapshot
The script also generates and maintains current-state.txt — a snapshot of every running service, open port, installed package, cron job, and disk usage. This serves as both documentation and a restoration target. If the rebuilt system doesn't match the snapshot, something was missed.
13. What's protected
Everything needed for full functionality: Docker services and their persistent data (including the MariaDB database for the book library). All 40+ custom scripts in ~/bin/. The complete beancount financial ledger. Vim configuration with custom key mappings (,j for journal, ,b for beancount, ,B for syntax check, ,r for reports). Bash configuration with aliases and functions. SSH keys for all remote access. OpenClaw configuration, session history, and memory. Git repositories with full history.
14. What could go better
The MariaDB backup is a filesystem copy, not a proper dump. This works because rsync runs at 4 AM when load is minimal and the database is small (155 MB), but a concurrent write during backup could produce a corrupted copy. A proper fix would be a pre-backup hook that runs mysqldump first. I haven't implemented this because the window for corruption is small and the data (book metadata) is recoverable from the books themselves if needed.
DNS propagation after an IP change can take up to 48 hours. The script can't control that. If speed matters, I'd need to have the cloud provider's API update DNS records as part of the bootstrap — which is scriptable but adds a dependency on API credentials being available outside the backup.