Rolled out a pilot of Moodle, an open-source learning management system, for one school branch within a larger multi-campus education group — the same self-hosted-first instinct behind the HR automation project, applied to the classroom side instead of HR records. This write-up covers the build, a couple of real near-misses, and the slow process of turning a set of empty named containers into an actually-correct roster.
The stack. Moodle deployed via Docker Compose (the app container, a PostgreSQL database, and Moodle's own background cron) on a dedicated VPS from a cloud provider, isolated from other infrastructure the same way the HR system's VPS is — a pattern that's becoming the default for anything holding real personal data belonging to the organization rather than personal infrastructure.
First pass at structure. Built a category tree mirroring the school's own hierarchy — a parent organization category, a branch category for the pilot campus, then one category per grade level, with one course shell per class section underneath. Each shell got a one-line summary (homeroom teacher, roster size) pulled from an existing attendance-tracking database already in use elsewhere for staff reporting.
The data-freshness problem, caught before it mattered. That attendance database turned out to be stale — months out of date, reflecting the previous school year's class assignments. A second, live source existed (an online registration system, current to the day) but only had grade-level data, no section/homeroom detail, and the school's academic year had just rolled over with a full promotion cycle — so the class-shell structure built from the stale source was almost certainly wrong for anyone it actually got used on. Caught this before any real student or teacher account existed, which made it a non-event rather than a cleanup job: nothing got bulk-imported until an actual current section list arrived directly from the school.
Branding, and a CMS surprise. Set the site logo and a short beta-announcement banner on the login page by writing a small one-off script that runs through Moodle's own bootstrap, using the same file-storage and config APIs its admin settings UI uses internally rather than touching the database directly. Updating the brand color later (to match the real logo rather than a generic placeholder blue) surfaced something unexpected: this Moodle release ships a built-in AI-generated login background image feature that overrides a plain brand color by default, complete with a small "AI-generated image" watermark — independent of, and silently overriding, the normal background-image admin setting. Had to explicitly suppress both the generated image and its watermark via CSS, on top of the actual color change. Also ran into Moodle's theme cache being revision-keyed rather than content-hashed: rapid edits within the same second can serve a stale cached stylesheet even after an explicit cache-clear, because the revision number didn't actually increment. Verified the fix by curling the compiled stylesheet directly rather than trusting a reload.
The bulk import. Once a real, current section list arrived from the school, ran a one-off script against Moodle's own user-creation and enrolment APIs (not raw SQL, to keep role assignments and enrolment records internally consistent) to create every teacher and student account at once — several dozen teachers with real school email addresses, and well over a thousand students who had no email or ID in the source file at all. Students got a placeholder email address and a sequential temporary ID standing in for their real school-issued student ID, a shared temporary password, and a forced password-change on first login. The sequential placeholder ID was deliberate: a stable join key to swap in the real student ID later without having to re-match names from scratch.
Role assignment doesn't work the way it looks like it should. Asked to grant all homeroom teachers broader access across the whole grade-level category (not just their own class), which led to a genuinely useful discovery: Moodle enforces actual course enrolment before a student can see anything, so a broad role grant for students is functionally inert — but for teachers, edit-capable roles cascade through context inheritance regardless of enrolment, so the same kind of broad grant gives real, live edit access to every class in the category, not just the teacher's own. Walked this through before it became a problem, and rolled the broad grant back to each teacher's own class only, once the asymmetry was clear. Also spent a while mapping where Moodle actually surfaces role-assignment screens for each scope (system-wide vs. category vs. course) — the category-level one in particular only appears on one specific page, not on the main course-management screen, something not obvious from the UI and only confirmed by reading the admin interface's own navigation source.
ID reconciliation, in rounds. The school sent real student IDs to replace the sequential placeholders, in a spreadsheet that was row-aligned with the original import order — useful, but not perfect: a meaningful chunk of rows had either a literal "not found" value or the same ID assigned to two different students (mostly sibling pairs, evidently a copy-paste error on the school's side). Flagged the problem rows back to the school rather than guessing, applied the clean majority, and left the rest on their placeholder IDs. Took three more rounds of corrected data from the school — each one closing most but not all of the remaining gaps, including one case where a freshly-issued "fixed" ID for one student collided with a completely different student's existing one — before every account finally carried its real school ID. One extra case surfaced during this process: a student who appeared on the school's own roster but had never been included in any data file sent so far, not a duplicate or typo, just a genuine gap in what had been exported. Created that one account directly rather than waiting on another file.
Restructuring from one course per class to one category per class. The original "one course shell = one class" model stopped making sense once actual subject-level courses needed to exist underneath each class rather than one container representing the whole thing. Converted every class from a single course into its own category containing one course per subject, carrying over its existing roster and teacher onto each new subject course rather than re-enrolling from scratch. Piloted the conversion on a single class first, got it checked against the live site before running the same logic as a batch across every remaining class. The batch run picked up one extra class that had quietly changed from hidden to visible between the pilot and the batch run, for reasons unrelated to this work — caught and flagged rather than silently absorbed, since it held a couple of pre-existing unrelated test accounts.
Real curriculum, replacing a four-subject placeholder. The initial per-class subject split used a generic stand-in list; once the school sent its actual curriculum breakdown by grade band, rebuilt every class's subject courses from that instead — different grade bands get different subject counts and names, with a couple of subjects deliberately merged or substituted depending on band per direct confirmation from the school rather than assumption. One subject area stayed intentionally empty of students (teacher-only) pending a separate roster that hadn't arrived yet at that point. The rebuild ran as one script per class, old placeholder subjects deleted and real ones created in the same pass, verified against the database directly afterward rather than trusting the script's own run-time output — it ran long enough that the process-monitoring layer briefly reported it as stalled when it had, in fact, finished correctly.
A human-readable course code. Added a structured code to every course — encoding school, grade, class section, and subject into one short identifier — after the first pass (just reusing the internal course shortname) wasn't the numeric scheme actually wanted. Exported the final mapping as a spreadsheet for the school's own reference.
A late roster, and a casing bug. A separate roster for one specific elective subject arrived later, covering a subset of classes rather than all of them. The first enrolment pass silently under-matched — a class-label parser produced mixed-case identifiers that didn't match the system's own all-caps naming convention for most classes, so only a small fraction of the roster actually landed before the mismatch was caught by checking the enrolled count against the expected one, not by any error being thrown. Fixed the casing and re-ran; the script was written to skip anyone already enrolled, so the partial first run caused no duplicates.
Where it stands now: every class in the pilot branch is fully structured by subject, every student and teacher account is reconciled against the school's real records with no outstanding placeholder IDs, and the elective roster is fully enrolled. Still open: a second school branch hasn't been started, and the subject-course model hasn't been tested yet against a grade band outside the pilot's own curriculum data.
]]>Mapped out a design for automating HR record-keeping for an organization with 400+ staff spread across multiple sites — currently split across scattered spreadsheets and email attachments, with no reliable way to track visa, labor card, or contract expiries. This is a design only; nothing below has been implemented yet.
The core decision is to not build custom. Frappe HR, an open-source HRIS on the Frappe Framework (Python, MariaDB, Redis for cache/jobs), already covers employee records, custom fields, notifications, and role-based field permissions — weeks of work to replicate from scratch. Deployment would use the official frappe_docker Compose stack (site + MariaDB + Redis + internal Nginx, managed via bench). Visa expiry, labor card expiry, and Emirates ID expiry become custom fields on the Employee doctype through the Customize Form UI — no code changes. Reminders ride on Frappe's built-in Notification doctype, triggered off those date fields N days out, able to call a webhook instead of just Frappe's own email/in-app alerts.
Hosting sits on a dedicated VPS, isolated from other infrastructure, with no public 80/443 exposed at the firewall level at all — reachable only over a WireGuard-based private overlay (the same protocol behind Twingate). That's network-layer isolation, not just an app login page. Data-at-rest encryption on the volume holding MariaDB's data directory sits on top of that as a second, independent layer.
Getting scattered data in has two paths. The base directory (name, branch, joining date) is a one-time bulk import through Frappe's own Data Import tool — CSV/Excel columns mapped to fields, no script needed. Expiry dates are the harder case, since they live on scanned documents, not structured text: the existing accounts-report pipeline already authenticates against the Microsoft Graph API and pulls email attachments for OCR, so the same auth flow and attachment handling can point at ID/visa scans instead, writing parsed dates to Frappe over its REST API (PUT /api/resource/Employee/<name>) rather than building a second ingestion path.
The part of the design that actually matters is the interface, because a web portal is only as good as people's willingness to log into it. Alerts and updates route through a WhatsApp group instead. Outbound: a scheduled job queries Frappe for records with an expiry inside the alert window and posts a message. Inbound: a plain-language reply (a visa renewal, a new joiner) gets parsed into a structured change, matched against the Employee list, and echoed back as a confirmation prompt — only an explicit reply triggers the actual write. Salary fields specifically sit behind Frappe's Role Permission Manager, scoped per role rather than per user, so the confirm-before-write step and the access model are two separate safeguards on the same sensitive data.
Threads worth pulling on further, none of it decided yet:
Trust, not just access. Confirm-before-write stops accidental bad parses, but it doesn't stop a bad actor with WhatsApp access. Salary changes specifically might warrant a second approver rather than a single self-confirm — and every change, regardless of size, should probably land in an immutable audit log (old value, new value, who confirmed, timestamp) rather than just overwriting the field.
Escalation, not just alerting. A single reminder N days out assumes someone reads it and acts. A tiered version — 30/14/7/1 days, escalating to a different person (branch admin → ultimately the owner) if nothing changes in the record after the first alert — turns "we sent a message" into "someone is actually accountable."
Keep the source document, not just the extracted date. Frappe supports file attachments per record. Attaching the actual scanned visa/labor card/Emirates ID to the Employee record — not just the OCR'd expiry date — means the source document is one click away when it's time to actually renew something, instead of a date with no paper trail behind it.
Cost forecasting. Every visa/labor card renewal has a government fee attached. A cost-per-renewal-type field, multiplied against the year's known expiries, turns this from a pure compliance tracker into a rough annual HR-compliance budget line — and it's a natural bridge to the existing beancount ledger rather than a fully separate system.
Onboarding and offboarding as workflows, not just directory edits. Frappe HR already ships Onboarding and Separation doctypes — checklists for what a new joiner needs (visa sponsorship steps, documents) and what a leaver triggers (clearance, final settlement). Worth using those directly instead of treating every hire/exit as a manual directory edit.
Retention policy. Once someone leaves, their salary and ID data doesn't just get deleted — labor law typically requires records be kept for a minimum period. Worth deciding upfront what "departed staff" looks like in the schema (archived vs. deleted, and for how long) rather than retrofitting it later.
The system itself needs a backup story. This is now infrastructure holding sensitive data on hardware separate from everything else — it needs its own place in the existing nightly backup routine, encrypted at the same standard as the live data, not an afterthought once it's running.
Updated Sep 29, 2026 18:48 UTC — a few more threads worth adding to the list above.
A dedicated number for the bot, not a personal-linked one. The WhatsApp interface should run on its own number, isolated from any personally-linked account, the same way a Telegram bot has its own separate identity. That keeps the compliance-sensitive group boundaries clean and avoids a scenario where confirmations for salary changes appear to come from a personal account rather than the system.
Don't trust every OCR read equally. The Graph-API-plus-OCR pipeline pulling expiry dates off scanned documents will occasionally misread a digit. Low-confidence extractions should route to manual review instead of flowing straight into a confirm-before-write prompt — a bad OCR read paired with a rubber-stamped confirmation is exactly how a wrong expiry date ends up as the system of record.
Push the audit log somewhere, don't just store it. An immutable change log is only useful if someone looks at it. A periodic digest — who's expiring soon, what changed since last time — timed to line up with the existing compliance committee cadence turns the audit trail into something that actually feeds that process, instead of a table nobody queries until something goes wrong.
Updated Sep 30, 2026 10:30 UTC — VPS sizing and the agent-hosting question, now resolved.
Sizing the VPS. Landed on a 4 vCPU / 8GB RAM NVMe-backed instance running Debian. At this scale the constraint isn't data volume — a few hundred employee records plus scanned documents is trivial — it's the number of always-on services a typical Frappe Docker Compose stack runs simultaneously: web server, several worker processes, a scheduler, cache, queue, and the database itself, each wanting its own slice of memory whether or not anything is actively happening.
A second encryption layer without a boot-unlock headache. Rather than full-disk LUKS encryption on the boot volume — which typically means entering a passphrase on every reboot, awkward for an unattended headless box — MariaDB's data directory and the attached scanned documents live on a separate network-attached block storage volume that the provider encrypts at rest by default (AES-256), no customer-managed key or boot-time unlock step required. That satisfies the "second independent layer" from the original design without adding an operational failure mode every time the box restarts.
Where the WhatsApp interface runs matters as much as where the database sits. The obvious shortcut was to add the WhatsApp bot as one more account on the automation gateway already running elsewhere, rather than standing up a second instance just for this. Wrong call: the isolation goal covers the whole conversational path, not only the database. Employee names, expiry dates, confirmation prompts, and the bot's own session credentials would otherwise flow through that other gateway's logs and its existing nightly backup routine — alongside unrelated personal data — which is exactly the mixing the dedicated VPS exists to prevent. Corrected design: the WhatsApp interface runs as its own agent instance, directly on the same isolated VPS as the database it talks to, with its own logs and its own backup path to the private backup target, independent of anything else.
Extending an existing agent, not inventing a third one. There's already a sandboxed, read-only business-automation agent running elsewhere with its own WhatsApp binding. Rather than designing a new agent identity from scratch for this, that existing one gets extended into the new environment: a fresh instance on the isolated VPS, a newly assigned WhatsApp number (not yet provisioned), and write access to Frappe's API layered in behind the confirm-before-write step described above — a capability upgrade on a known, already-sandboxed agent rather than a fourth thing to reason about.
Updated Oct 08, 2026 UTC — the design moved from concept to a running system over the following days.
From design to a running system. Deployed on the isolated VPS described above, running in Docker. The employee directory came in from several spreadsheets, one per site, each with its own column layout — so the import needed per-sheet header detection rather than one fixed column mapping. Recovering expiry dates from the same source required reading formula-evaluated values rather than raw cell contents, since several sheets computed dates via formulas instead of storing a literal date.
Salary reconciliation, done in passes. Matching a separate payroll export against the directory went exact-match first, then fuzzy name-matching with manual confirmation on anything ambiguous, then a one-by-one pass on what was left — new joiners not yet in the directory, or departed staff being incorrectly reintroduced. A batch-creation bug during the original import re-scanned entire sheets instead of only the unmatched rows, creating a wave of duplicate records mixed in with genuinely new ones. Cleanup ran in two passes: a creation-timestamp cutoff plus loose token-matching for the bulk of it, then a string-similarity pass to catch the near-duplicates — typos, spacing differences, OCR-style variants — the first pass missed. Where a duplicate carried data the original lacked, that data got merged in before the duplicate was deleted, never the reverse.
Matching gets harder than exact-or-fuzzy. A later reconciliation round needed two refinements: de-duplicating repeated name tokens before scoring similarity, and falling back to first-and-last-name matching when a record has no middle name rather than failing the comparison outright. A dedicated check — independent of the overall similarity score — caught a same-surname, different-person collision that would otherwise have applied one person's pay figure to someone else's record; the overall score alone scored it high enough to pass. A handful of genuine gaps, people absent from payroll data across two consecutive months, went to manual follow-up rather than being force-matched at a lower confidence threshold — in keeping with the original design principle of flagging low-confidence results instead of auto-applying them.
Two payroll snapshots, not one replacing the other. A later month's payroll file used a different column layout than the first — no net-pay column; gross had to be computed from basic pay plus allowances instead — so rather than overwrite the first month's figures, both now sit side by side as separate snapshots, useful for cross-checking one month against another rather than trusting either in isolation.
Housekeeping on the record layout. Reorganized the custom fields so salary and employment-status data live under their logical sections instead of one catch-all block, and hid — not deleted — the fields that weren't part of daily use. The reordering needed the platform's own form-customization mechanism rather than a direct field edit; a direct edit silently misplaced one block while appearing to work on another.
The interface needed its own isolated agent, not just its own database. Resolved the open question from the design above: rather than adding one more account to the automation agent already running elsewhere, the messaging interface now runs as a fully separate agent instance, directly on the isolated VPS, through its own subscription login with a different LLM provider than the main instance — different enough to count as a genuinely separate environment, not a second account on a shared one. It has its own persona, deliberately exacting and procedural in tone given what it handles, and inherited the standing caution rules (read-only on sensitive config, prefer-safe-delete, ask-first on infrastructure changes) from the main assistant's own rulebook, for consistency.
Messaging landed on a different platform than planned first. WhatsApp pairing got as far as a scannable QR code before being deferred. In the meantime, a Telegram bot connection went live instead — far simpler to wire up without an interactive pairing flow — so the alerts below are flowing there today, with WhatsApp still the eventual target.
The confirm-before-write mechanism, for real. A small queue now sits between the agent and the database: a proposed change gets recorded but not written, requires one confirmation for most fields and two independent confirmations for salary changes specifically, and only writes through the API once that threshold is met. Tested end-to-end against real records, including the two-confirmation salary path, before going live — a genuine gate the agent can't bypass unilaterally, not just an instruction it's been told to follow.
Alerts and a weekly digest, both live. Tiered reminders (30/14/7 days out, plus already-expired) run daily across every tracked expiry type, de-duplicated so the same alert doesn't repeat but does escalate if it crosses into a more urgent tier. The first real run surfaced a meaningful backlog of already-expired records, some stale for years — reported plainly rather than softened, which is the tone the whole system is built around. A weekly digest adds compliance counts, pending-queue activity, and a genuine audit trail pulled from the platform's own built-in version history rather than a separately bolted-on log.
Still open: an escalation contact beyond the primary reviewer, a private network path to reach the new instance's dashboard remotely, a backup routine for this instance's own data, and the WhatsApp migration itself.
]]>A series of maintenance tasks across the homelab this week. The main items were getting the book library fully operational after a migration, auditing the backup chain, and cleaning up the monitoring setup. Nothing was on fire — this was preventive work.
1. Book library migration
The book library (Grimmory) was moved to server2 as a Docker container backed by MariaDB. The migration involved pulling the entire book collection from server1's backup storage — about 632 MB of ebooks and comics. The container picked up most books automatically, but comics in .cbz format needed their permissions fixed before the library scanner would index them.
The MariaDB database holds all metadata: reading progress, tags, collections, and cover art cache. Initially excluded from backups to avoid complexity, but it's only 155 MB and rsync handles it fine. Added it to the nightly backup — won't break the running database because rsync only reads, and at 4 AM the DB is idle.
2. Document management
Set up Papra for document management at docs.example.com. Another Docker container, this one simpler — just needs a data directory for uploaded files. Caddy handles the reverse proxy and TLS termination. The DNS record was added to the zone on server1 and propagated to server2 via AXFR.
3. Monitoring consolidation
Uptime Kuma was set up at kuma.example.com on server2. It monitors all public services — the book library, document manager, and the main website. It also monitors server1's web endpoint to verify it's serving pages. Alerts go to Telegram. The container runs as user 1000:1000 (non-root) as part of the security hardening done earlier.
4. Backup chain audit
Reviewed the full backup topology. Two daily crons on server2: backup-books-to-server1 at 04:00 (books only, with --delete) and backup-sync at 04:10 (entire home directory, without --delete). The books get backed up twice — once standalone, once as part of the full sync. Redundant but harmless. The standalone job uses --delete to remove books from server1 that were deleted locally; the full sync doesn't, which means deleted files persist on server1 until manually cleaned.
Total backup footprint on server1 is about 1.7 GB. Backup logs go to ~/journal/homelab-backup/ with dated filenames for easy review.
5. CalDAV status
Radicale on server1 is running again after the last pkg_add -u update. It handles CalDAV and CardDAV for calendar and contact sync. The service runs behind relayd on port 5232. No issues after the package update — just needed a service restart.
6. Homelab health snapshot
Current state of all machines: the router is stable (25 days uptime, low load). Home Assistant running clean (7 days since last restart, 42% disk). The NAS root disk is at 85% — worth monitoring but not critical since the data volumes have plenty of space. The MacBook is at 58 days uptime with only 6% disk usage. Both VPS instances healthy with low load.
7. What's next
The NAS disk usage needs attention before it hits 90%. The duplicate book backup job could be consolidated into the main sync with --delete, but that's a low-priority cleanup. The monitoring setup is complete — any service going down triggers a Telegram alert within 60 seconds.
I run two VPS instances. server1 is an OpenBSD box that serves static files and acts as my secure anchor. server2 is a Debian machine that runs containers and does the heavy lifting. The goal was to build a DNS architecture where server1 is the hidden master — it holds the authoritative zone file but never appears in public NS records. Public queries hit server2 and a cloud provider's secondary nameserver. server1's IP stays out of DNS entirely.
1. The architecture
server1 runs NSD as the hidden master. The zone file lives at /var/nsd/zones/master/ and contains all records for the primary domain. server2 also runs NSD, configured as a secondary that pulls from server1 via AXFR (zone transfer). A cloud provider acts as the third nameserver, synced manually with a script.
The nameserver delegation at the registrar points to ns1.primarydomain (server2) and the cloud provider's NS. server1 is nowhere in the public chain — it only speaks to server2 over a private AXFR channel.
2. NSD configuration on server1
NSD on OpenBSD is straightforward. The config at /var/nsd/etc/nsd.conf defines the zone and allows AXFR transfers only to server2's IP. The zone file follows standard RFC 1035 format with SOA, NS, A, CNAME, and MX records. Every edit requires incrementing the serial number — I use the YYYYMMDDNN convention so it's always monotonically increasing.
After editing, doas rcctl reload nsd picks up the changes. server2 polls for SOA serial changes and triggers AXFR automatically when it detects a higher serial.
3. NSD configuration on server2
server2's NSD is configured as a secondary with request-xfr pointing to server1. It stores the received zone in /var/lib/nsd/. The refresh interval is set to 3600 seconds (1 hour), but NOTIFY messages from server1 trigger immediate transfers for most changes.
4. Cloud provider sync
The cloud provider doesn't support AXFR inbound, so I wrote ~/bin/sync-dns-to-cloud — a shell script that reads the zone file, parses it, and pushes records via the provider's API. It handles A, AAAA, CNAME, MX, and TXT records. Not ideal, but it only needs to run when I change the zone, which is rare.
5. Zone management workflow
The workflow for any DNS change is: (1) SSH to server1, (2) edit the zone file and increment the serial, (3) doas rcctl reload nsd, (4) server2 picks it up via AXFR within seconds, (5) run the sync script for the cloud provider. Three steps if I skip the cloud sync for records that don't need geographic redundancy.
6. DNSSEC signing
After the base DNS was working, I signed the zone with DNSSEC. Generated both a Zone Signing Key (ZSK) and a Key Signing Key (KSK) using ldns-keygen with ECDSAP256SHA256 (algorithm 13). The ZSK signs individual records, the KSK signs the DNSKEY RRset.
The signing process: ldns-signzone takes the unsigned zone file and both keys, outputs a signed zone with RRSIG, NSEC, and DNSKEY records. I then published the DS record at the registrar — this is the trust anchor that chains the domain to the root zone. The DS record contains the KSK's key tag, algorithm, digest type, and digest hash.
7. Verification
Tested with dig +dnssec to confirm RRSIG records are returned, delv for full chain-of-trust validation, and online tools like DNSViz to visualize the chain. All checks passed — the zone validates from root to leaf.
8. Why this architecture
The hidden master pattern keeps server1's IP out of public records entirely. Anyone querying DNS sees only server2 and the cloud provider. server1 is invisible to scanners, bots, and anyone enumerating the domain. This matters because server1 also handles other sensitive services — the less exposure, the better.
Geographic redundancy across two physical locations and a cloud provider means DNS stays up even if one goes down. And since server1 is the single source of truth, there's never a split-brain problem — changes flow in one direction only.
9. Edge cases handled
If server1 goes down, server2 and the cloud provider continue serving the cached zone until the SOA expiry (set to 1 week). If server2 goes down, the cloud provider handles all queries. If both secondaries go down, DNS is offline but server1 is untouched and can be re-synced once connectivity returns.
The AXFR channel between server1 and server2 is restricted by IP in NSD's config — only server2's IP can request transfers. This prevents unauthorized zone dumps.
10. What's not included
I didn't set up automated DNSSEC key rotation. The current keys work indefinitely, but best practice is to rotate the ZSK quarterly and the KSK annually. That's a future task. I also didn't automate the cloud provider sync — it's manual because DNS changes are infrequent and I want to review each one before it goes live.
]]>Both servers were running stock SSH configs. Not terrible, but not locked down either. The goal was to eliminate password auth entirely, restrict which users can log in, rate-limit connection attempts, and close every port that doesn't need to be open. Two different operating systems meant two different approaches to the same problem.
1. server1 (OpenBSD) — SSH config
OpenBSD ships with a conservative sshd by default, but I tightened it further. In /etc/ssh/sshd_config: PasswordAuthentication no, PermitRootLogin no, AllowUsers sysadmin (only one user can SSH in), MaxAuthTries 3, ClientAliveInterval 300 (drop idle connections after 5 minutes), and all forwarding disabled — AllowTcpForwarding no, AllowAgentForwarding no, X11Forwarding no.
2. server1 — pf rate limiting
OpenBSD's pf firewall handles connection-level rate limiting. I added a <bruteforce> table and a rule that allows SSH connections with max-src-conn-rate 5/60 — if any IP opens more than 5 connections in 60 seconds, it gets added to the bruteforce table and all further connections are dropped. The table is persistent across reboots.
The pf rules also explicitly block mail ports (25, 587, 993) since server1 no longer runs mail services. Only ports 80 and 443 are open for web traffic, plus 53 for DNS.
3. server1 — the lockout incident
During initial testing of the pf rate limiting, I hit a problem. The daily backup script from server2 opens multiple SSH connections to server1 in quick succession — rsync, status checks, config pulls. This triggered the rate limiter, and server2's IP got added to the bruteforce table. server2 could no longer reach server1 at all.
I had to access server1 through the cloud provider's web console (out-of-band management), reboot the machine to flush the pf tables, and then fix the config. The solution was adding server2's IP to a whitelist that gets evaluated before the rate limit rule. Order matters in pf — rules are evaluated first-to-last, and the first match wins.
4. server2 (Debian) — SSH config
Same sshd hardening as server1: key-only auth, no root login, AllowUsers user, MaxAuthTries 3, ClientAliveInterval 300. The difference is in the forwarding: server2 allows TCP forwarding because Docker needs it for container networking. Everything else is disabled.
5. server2 — ufw + fail2ban
Debian doesn't have pf, so rate limiting uses two layers. ufw with LIMIT on port 22 handles basic connection throttling — it drops connections from IPs that attempt more than 6 connections in 30 seconds. On top of that, fail2ban watches /var/log/auth.log for failed login attempts and bans offending IPs for 10 minutes after 5 failures.
The fail2ban stats tell the story: 435 total bans since installation, with 2 IPs currently blocked at any given time. Most are automated scanners hitting port 22 and trying common usernames. They get exactly 5 attempts before they're firewalled.
6. Docker non-root hardening
While auditing the security posture, I noticed Uptime Kuma was running as root inside its container. Changed the docker-compose to run as user 1000:1000. This means even if someone exploits a vulnerability in the monitoring service, they can't escalate to root inside the container, and container breakout gives them an unprivileged user on the host.
7. Automated config backups
Created ~/bin/backup-configs that runs daily at 2 AM via cron. It copies SSH configs, pf rules (server1), ufw rules (server2), Caddy config, NSD zone files, and cron tabs to a timestamped directory. Retention is 30 days — older backups get deleted automatically. If a config change breaks something, I can diff against yesterday's backup and roll back.
8. Automatic security patches
server1 runs /usr/local/bin/security-updates weekly on Monday at 3 AM. It applies syspatch (OpenBSD binary patches) and runs pkg_add -u (package updates). server2 has unattended-upgrades configured for security-only updates. Both log their output for review.
9. File integrity monitoring
AIDE (Advanced Intrusion Detection Environment) is set up on server2. It generates a baseline database of file hashes for critical system directories — /etc, /usr/bin, /usr/sbin. Daily checks compare current state against the baseline and report any unexpected changes. This catches unauthorized modifications to system binaries or configs.
10. Firewall audit results
Final port inventory after hardening. server1: 22 (SSH, rate-limited), 53 (DNS), 80/443 (web). Mail ports 25, 587, 993 explicitly blocked. server2: 22 (SSH, ufw LIMIT + fail2ban), 80/443 (web, Caddy reverse proxy), Docker bridge ports for internal container communication only. Both servers — no unnecessary services listening, no world-accessible management ports, no password auth anywhere.
11. What's not included
I didn't set up intrusion detection on server1 (OpenBSD has its own integrity checking via mtree, which I haven't configured yet). I also didn't implement SSH certificate auth — key-based is sufficient for two servers with one admin each. Port knocking was considered and rejected as unnecessary complexity given the rate limiting already in place.
The question was simple: if server2 dies tomorrow — hardware failure, provider goes down, accidental destruction — how fast can I rebuild it from zero? The answer before this work was "hours of manual setup, hoping I remember everything." The answer now is one script and about 20 minutes of automated restoration.
1. The problem
server2 runs the entire operational stack: Docker containers for a book library, document management, and monitoring. A Caddy reverse proxy with automatic Let's Encrypt certificates. 40+ custom shell scripts. Cron jobs for backups, reports, and maintenance. OpenClaw for automation. Git repos for journal and financial ledger. Vim config with custom key mappings and beancount shortcuts. All of it backed up nightly to server1, but no automated way to put it all back together.
2. The backup system
A daily cron at 04:10 UTC runs rsync from server2 to server1, syncing the entire home directory to /mnt/vps-backup/. Excludes are minimal — just package caches and virtualenvs that can be regenerated. A separate cron syncs the book library at 04:00. Total backup size is around 1.7 GB. This means server1 always has a <24 hour old copy of everything on server2.
3. What the bootstrap script does
The script assumes a fresh Debian VPS with root access and an SSH key that can reach server1. It runs in sequence:
4. System packages
Installs everything server2 needs: Docker and docker-compose, Python 3 with pip, Node.js (for OpenClaw), git, rsync, vim, fail2ban, ufw, and build essentials. Python packages include beancount, whisper, and their dependencies. The package list was generated by diffing a fresh Debian install against server2's current state.
5. Pull from backup
rsync pulls the entire /mnt/vps-backup/ from server1 back to the new machine's home directory. This restores all scripts, configs, dotfiles, journal, financial ledger, memory files, and application data in one shot. The SSH key to server1 is the only prerequisite — everything else flows from the backup.
6. Restore Docker services
The docker-compose files are part of the backup, so restoration is: navigate to each service directory and run docker-compose up -d. The book library needs its MariaDB data directory (included in the backup), the document manager needs its uploaded files (included), and the monitoring tool needs its SQLite database (included). Caddy config is also in the backup — it picks up the reverse proxy rules automatically.
7. Restore cron jobs
A crontab.backup file is maintained in the home directory and synced to server1 daily. The script loads it with crontab crontab.backup. This restores all 14 scheduled jobs — backup syncs, daily digests, school reports, financial reports, memory maintenance, and journal sync.
8. Fix git remotes
The journal and finance repos have origin URLs pointing to GitHub. After restoration, git pull/push works immediately because the SSH keys (for GitHub) are part of the home directory backup. The script verifies each repo can reach its remote.
9. Install OpenClaw
OpenClaw is installed via npm. The config, session history, cron state, and memory files are all in the backup. After installation, the agent resumes with full context — it knows the infrastructure, the user's preferences, and all scheduled tasks.
10. Validation
A separate validation script runs 44 checks across every component: Docker services responding on expected ports, cron jobs loaded, SSH key access to server1 working, git repos in clean state, all 40 bin scripts present and executable, Caddy serving HTTPS, DNS resolving correctly, fail2ban running, backup cron scheduled. Each check prints pass/fail with details.
11. Pre-restore checklist
A markdown checklist documents what to do before running the script: provision a new VPS at the same provider, ensure the SSH key to server1 is available (it's also backed up to an encrypted local copy), update DNS if the IP changes, and verify server1's backup is current.
12. Current system snapshot
The script also generates and maintains current-state.txt — a snapshot of every running service, open port, installed package, cron job, and disk usage. This serves as both documentation and a restoration target. If the rebuilt system doesn't match the snapshot, something was missed.
13. What's protected
Everything needed for full functionality: Docker services and their persistent data (including the MariaDB database for the book library). All 40+ custom scripts in ~/bin/. The complete beancount financial ledger. Vim configuration with custom key mappings (,j for journal, ,b for beancount, ,B for syntax check, ,r for reports). Bash configuration with aliases and functions. SSH keys for all remote access. OpenClaw configuration, session history, and memory. Git repositories with full history.
14. What could go better
The MariaDB backup is a filesystem copy, not a proper dump. This works because rsync runs at 4 AM when load is minimal and the database is small (155 MB), but a concurrent write during backup could produce a corrupted copy. A proper fix would be a pre-backup hook that runs mysqldump first. I haven't implemented this because the window for corruption is small and the data (book metadata) is recoverable from the books themselves if needed.
DNS propagation after an IP change can take up to 48 hours. The script can't control that. If speed matters, I'd need to have the cloud provider's API update DNS records as part of the bootstrap — which is scriptable but adds a dependency on API credentials being available outside the backup.
]]>I've been tracking personal finances in beancount for a while now, but the tooling around it has evolved significantly. The ledger file lives in a private git repo, and I've built a set of scripts and automation around it that make the whole system practical for daily use.
1. The ledger
Everything lives in ~/finance/accounts.beancount — a plain-text double-entry ledger. Operating currency is AED, with CAD positions tracked separately for a Canadian brokerage. The file follows beancount's syntax: account declarations at the top, then chronological transactions. Each transaction has a date, payee, narration, and at least two postings that sum to zero.
2. Account structure
The chart of accounts covers: checking accounts (AED and CAD), credit cards, brokerage accounts (AED equities, CAD positions in three funds), a family member's managed portfolio, and expense categories broken down by type — groceries, dining, transport, subscriptions, services, and so on. Income tracks salary and investment returns separately.
3. Vim integration
Since I edit the ledger in Vim, I set up key mappings and abbreviations to speed up entry. ,b opens the ledger, ,B runs bean-check for syntax validation, ,r runs the report script. Abbreviations expand common accounts: expg becomes Expenses:Food:Groceries, ac becomes Assets:Bank:Checking, lcc becomes Liabilities:Credit:Card. These are defined in .vimrc and make manual entry fast enough that I don't need a separate input tool.
4. Reporting scripts
Two main scripts handle reporting. ~/bin/networth is the original Python script that parses the ledger and outputs a detailed breakdown — assets by account, liabilities, net position, and month-over-month change. ~/bin/networth-report is a newer script that generates markdown reports for automated delivery. It combines net worth and budget data into a single document.
The budget section compares actual spending against caps defined in the script. Categories with caps: groceries (3,000), dining (3,000), services (2,000), subscriptions, gas, personal care. When spending exceeds a cap, the report flags it. The total budget target is around 19,000 AED/month.
5. Automated monthly reports
Three cron jobs generate reports: on the 1st (previous month final), the 15th (mid-month check), and the last day (month-end snapshot). Each runs the report script and sends the output to Telegram. This means I get a financial summary delivered three times a month without doing anything.
6. Statement reconciliation
When bank statements come in, I diff them against the ledger to find discrepancies. This has caught missing transactions — a gas station charge here, an interest charge there. The process is manual but fast: export the statement, compare line by line, add anything missing. Beancount's bean-check validates the result.
7. Multi-currency handling
The AED ledger is straightforward, but the CAD positions add complexity. Beancount handles multiple currencies natively — each posting specifies its currency, and the price directive sets exchange rates for net worth calculations. The CAD brokerage holds positions in three funds, each tracked as a separate commodity. Monthly investment targets are defined separately and checked against actual contributions.
8. Git versioning
The finance directory is its own git repo, pushed to a private remote. Every edit gets committed — the gitpush function in .bashrc handles journal, finance, and dotfiles in one command. This gives full history of every transaction ever added, modified, or corrected. If I fat-finger an amount, git diff shows exactly what changed.
9. What works well
Plain text means the ledger is greppable, diffable, and version-controlled. No vendor lock-in, no subscription, no cloud dependency. The Vim shortcuts make entry as fast as any GUI. Automated reports mean I actually look at the numbers regularly instead of ignoring them. The git history provides a complete audit trail.
10. What's not included
I don't use fava (the web UI for beancount) — the terminal workflow is sufficient for now. I don't have automated bank import — transactions are entered manually, which is actually a feature since it forces me to review each one. I don't track individual stock lots for tax purposes since the jurisdictions I operate in don't have capital gains tax on securities.
]]>