SYS://FIELD-REPORT-09

The Backup You Never Restored Is a Guess

The app worked. It looked finished. The auth was even solid — the owner had done the hard part everyone skips. And it was still one bad DROP TABLE, one corrupted migration, one fat-fingered rm -rf away from gone, with nothing behind it. Not a stale backup. Not an old one. None. Here’s the teardown, worst finding first.

This is a representative audit — the same shape I hit over and over on apps built with Lovable, Bolt, v0, Cursor, and Claude Code. Names and paths are changed, but every finding is real-shaped and drawn straight from the report the launchworthy skill produces. Call the app ledgerly: a SvelteKit invoicing tool on a self-hosted Postgres box, one VPS, live, with a handful of paying customers already entering real financial data.

Auth passed. The interesting failure was somewhere nobody looks.

1/5
domains passing
0
critical findings
5
high findings

Zero criticals. Read that as good news and you’ve missed the point. ledgerly has no gaping RLS hole, no service key in the bundle — the loud, screenshot-worthy failures aren’t here. What’s here is quieter and, for a live app with customer data, arguably worse: the whole of Operations & Recovery is a FAIL, and the single worst finding is only a [HIGH]. That’s the honest, uncomfortable shape of this one. The thing that ends the app isn’t a critical. It’s three highs and a medium that only matter on the worst day, which is exactly why they never got built.

The finding that ends the app: no backups, no restore, no rollback

On a self-hosted Postgres box, backups are not a checkbox the platform ticks for you. Supabase, Neon, and PlanetScale run automated backups you can lean on. A raw Postgres install on a VPS backs up nothing until you write the cron job yourself. ledgerly never did.

I checked the obvious places. No pg_dump in a crontab. No backup script in the repo. No object-storage bucket filling up nightly. The deploy/ folder had a single docker compose up -d and nothing else. The database on that one disk is the company. Lose the disk, lose the customers.

CRITICAL · CAUGHT

No automated backups on a self-hosted Postgres holding live customer data

Every invoice, every client record, every dollar figure lives on one Postgres volume with nothing copying it anywhere. A bad migration, a disk failure, or one wrong command and it is all gone — permanently, with no version to fall back to. The app keeps looking finished right up until the second it doesn’t exist.

Per the launchworthy checklist this is a [HIGH], not a [CRITICAL] — and I’m going to hold that line even though the blast radius feels catastrophic, because the severity tags are load-bearing and inflating them for drama is how audits lose trust. But it’s a [HIGH] stacked on two more that make it worse. No documented or built-in way to roll back a bad deploy is a second [HIGH]: self-hosted Docker with no previous image retained means a broken release has no undo. And backups never restore-tested is a [MEDIUM] that would apply the moment the first two are fixed — because a backup you have never restored is not a backup. It’s a guess.

The fix isn’t clever. It’s just doing the thing before you need it.

THE FIX · IN ORDER

Make recovery exist before the incident

  1. Automate the dump. A nightly pg_dump to off-box object storage (S3, R2, B2), on a cron, with retention. The database must live somewhere other than the machine that can die.
  2. Retain the last good image so a bad deploy has an undo. Tag releases, keep the previous container image, and write the two-line “roll back to the last tag” command down where you’ll find it at 2am.
  3. Restore it once, on purpose. Spin up a throwaway box, pg_restore last night’s dump into it, and confirm the row counts match. Until you’ve done this, you don’t have a backup — you have a file you hope is a backup.

Verify it by hand This is a MANUAL CHECK — code can’t prove your backups run or restore. Trigger a dump, then actually restore it into a scratch database and diff the row counts against production. Evidence the fix landed: a restored table you can query, and a screenshot of the object-storage bucket with last night’s file in it, timestamped.

The false alarm: “I’ll deal with backups if something breaks”

Here’s the credibility move, and the reason to trust an audit that gets this right instead of just piling on.

The instinct — and it’s a reasonable-sounding one — is that backups are an incident-response tool. Something breaks, you go get the backup, you restore, you move on. So why build it now, pre-incident, when there’s a launch to ship?

FALSE POSITIVE

Backups are a thing you can set up later, after something goes wrong

This is exactly backwards. The moment something breaks is the one moment you cannot make a backup — the data you’d want to save is already the corrupted, deleted, or lost data. Recovery is not incident response. It is the one piece of infrastructure that has to exist strictly before the incident, or it does not exist at all. “I’ll deal with it if something breaks” is not a plan for backups. It’s a plan for not having them, phrased optimistically.

Every other item on a punch list can, in principle, wait for the thing to go wrong and then get fixed. A missing loading state annoys someone; you add it. A slow query drags; you index it. Recovery is the sole exception. It is the one category where “later” and “never” are the same word, because the deadline is set by an event you don’t control and won’t see coming. That’s what makes it the spine of this report: it’s the finding that punishes procrastination the hardest, on the app where auth was already done right.

The rest of the punch list

Auth passed and the app is genuinely well-built, so the list is short. It’s ranked worst-first so the scope stays honest — three of these are the recovery story above, the rest is real but not existential.

FIX THIS WEEK

After you have a tested backup

  1. [HIGH] No automated database backups on the self-hosted Postgres. This is finding one — a nightly pg_dump to off-box storage with retention.
  2. [HIGH] No documented or built-in way to roll back a bad deploy. Retain the previous image; write the rollback command down.
  3. [HIGH] No error tracking. No Sentry, no handleError hook in src/hooks.server.ts. Production throws into the void and you learn it broke from a customer, not a system.
  4. [MEDIUM] Backups never restore-tested — moot until findings 1 and 2 land, then non-negotiable. Restore once and diff the row counts.
  5. [MEDIUM] No uptime monitoring. A free UptimeRobot check on a /healthz route tells you the box is down before your customers do.

Two MANUAL CHECKs round it out, and they’re the kind code can never surface. First: the four-part “rebuild in under an hour” test — code in git (yes), env vars in a password manager (yes, they were in a .env on the VPS and also in the owner’s Bitwarden), deploy steps written down (partially — the docker compose line existed, the rollback didn’t), and the database restorable from backup (no). Name which of the four is missing and you’ve named your single point of failure. For ledgerly it’s the fourth leg, cleanly. Three of four solid is not three-quarters safe; it’s one missing leg on a stool.

Second: domain registrar 2FA. Losing the domain loses everything downstream of it — email, DNS, the app’s front door — regardless of how good your Postgres backups are. Confirm the registrar has 2FA on and a recovery email you actually control. It’s a two-minute check that sits under the entire business.

ASK YOURSELF

“I’ll deal with backups if something breaks.” The moment something breaks is the one moment you cannot make a backup — the data you’d want is already gone. Recovery is the one thing that has to exist before the incident, not after. And an untested backup counts as no backup until you’ve restored from it once.

Why this keeps happening

None of this means ledgerly’s owner did anything foolish. The opposite — they got auth right, which is the part most people botch. The gap is structural. AI builders, and honestly most tutorials, are extraordinary at getting you to looks finished: a working app, a clean UI, a login that holds. They are silent about recovery, because recovery is invisible on your own screen. On day one there’s nothing to restore, nothing has broken, and the backup you don’t have looks identical to the backup you do — right up until the day they look very different.

That’s the gap launchworthy is built to stand in. It plays bouncer at the door of production: it counts a backup you’ve never restored as no backup, it asks whether you can actually roll back tonight’s deploy, and it refuses to mark Operations & Recovery green just because the code compiles. The code was never the problem here. The problem was that the worst day hadn’t happened yet — and recovery is the one thing you can only build before it does.