SYS://FIELD-REPORT-15
The Monitor Was Green Through the Whole Outage
The monitoring box was ticked. There was a /health endpoint, and there was an uptime monitor pointing at it, reporting green. On paper, operations was handled. Then I read the two pieces. The health endpoint’s entire body was return new Response("ok") — it answered 200 whether or not the database was alive. And the monitor was a scheduled function inside the same deployment it was watching, so if the app went down, the thing checking it went down with it. Both would report calm straight through a real outage. A monitor you can’t trust is worse than none, because it’s the reason nobody looks.
This is a representative audit — the exact pattern I hit over and over on apps built with Lovable, Bolt, v0, and Cursor. Names and paths are changed, but every finding below is real-shaped and drawn straight from the kind of report the launchworthy skill produces. Call the app roomboard: a SvelteKit app on self-hosted Postgres, health check present, monitor configured, about to go live.
Nothing here throws an error, and that’s the trap. Both failures are green-when-it-should-be-red, which is the one kind of failure a dashboard actively hides.
The finding: a health endpoint wired to nothing
A monitor reads a status code and nothing else. So the whole job of a health endpoint is to return a non-2xx when the app is actually unusable. roomboard’s returned 200 unconditionally:
// routes/health/+server.ts
export function GET() {
return new Response("ok") // proves the process is up. Nothing else.
}
This tells you the Node process is running and can serve a route. It cannot see a dead database, an expired third-party key, or a queue that stopped draining — the outages people actually have. It returns 200 straight through all of them.
HIGH · CAUGHT
Health endpoint returns a static 200
routes/health/+server.ts checks nothing and always answers 200. A no-op health check is a [MEDIUM] on its own; here it’s the only signal wired to the uptime monitor, which makes it an active false green — the dashboard is confidently green because the check is incapable of being anything else. Raised to [HIGH].
The fix: touch the dependencies the app can’t work without, and let the status code carry the verdict.
THE FIX
// routes/health/+server.ts
import { json } from "@sveltejs/kit"
import { db } from "$lib/server/db"
export async function GET() {
const checks: Record<string, boolean> = {}
try {
await db.query("SELECT 1") // cheapest round trip that proves the DB answers
checks.database = true
} catch {
checks.database = false
}
const healthy = Object.values(checks).every(Boolean)
return json({ status: healthy ? "ok" : "degraded", checks },
{ status: healthy ? 200 : 503 }) // the status code is the signal
}Three rules for any stack: return a non-2xx (503) when a dependency is down — a 200 body that says { db: false } is invisible to every monitor; keep it out of the cache (mark it dynamic, send Cache-Control: no-store) so it reports the app’s state, not the build’s; and don’t dump secrets or a full diagnostic into a public body. Don’t check things the app can survive without either — if a broken analytics provider makes /health return 503, you’ve built a pager that fires for nothing, and pagers that fire for nothing get muted.
The second finding: the watcher lives inside the patient
The monitor was a scheduled job in the same project as the app. When the host, the region, the DNS record, the TLS certificate, or the deploy itself is what breaks, that job is down too — and reports nothing, which reads as silence, not as an outage.
MEDIUM · CAUGHT
Monitoring runs inside the thing it monitors
roomboard’s uptime check is a cron in the same docker-compose.yml as the app. It asks the server whether it feels okay. The outages that matter are exactly the ones that take the checker down with the app, so it can only report green. At least one check has to originate outside your infrastructure.
THE FIX
Put a free external monitor in front of the honest health endpoint — UptimeRobot, Better Stack, Cronitor, Pingdom, or your host’s. What matters is that the check originates on someone else’s network. Configure it to hit the production URL users actually type (through the CDN/proxy they hit, not the origin or a preview), every 1–5 minutes, alerting a human by email/Slack/SMS after 2 consecutive failures. Turn on certificate-expiry alerting if offered — an expired TLS cert takes the whole site down with no other warning. Then break it on purpose: stop the database and confirm the monitor goes red and the alert reaches you. An untested monitor is an assumption.
The gap under both: nothing watches the critical path
Even a perfect uptime check is only the floor. The homepage can return 200 for weeks while signup throws on the insert, or the core booking action returns empty. A ping can’t see any of that.
THE FIX · SYNTHETIC
Pick the one flow roomboard exists to do — creating a booking — and run it end to end against production on a schedule. If Playwright is already in the repo, a spec that signs in, books, and asserts the confirmation, run from CI on a schedule: cron (not just on push), is nearly free. No CI? Better Stack / Checkly / Datadog run scripted browser checks on a schedule. Minimum viable: an authenticated /api/health/deep that performs the real write path against a dedicated test account and rolls it back. Use a dedicated synthetic account, prefix its records so they’re identifiable, and exclude them from analytics and billing — a synthetic check that pollutes your metrics gets deleted within a month.
The false alarm: “the dashboard is green, so we’re up”
This is the belief the whole setup was built to produce, and it’s the dangerous one.
FALSE POSITIVE
⚠️ The monitoring dashboard is green, so the product is working.
Green means a URL answered. That’s all it can mean. It does not mean the database is alive (the endpoint returns 200 regardless), it does not mean the app is actually reachable by users (the checker lives inside the deployment), and it does not mean anyone can complete the thing the app is for (nothing exercises signup or booking). Presence of a monitoring account is not coverage. A check that runs from outside and fails when the app is broken is coverage — and until that exists, the green light is just the reason the outage runs longer before a human notices.
Verify by hand
Break a required dependency — stop Postgres — and confirm /health now returns a non-2xx and the external monitor goes red and pages you. Confirm the monitor’s target is the production host users reach and its most recent check is minutes old, not weeks. Then name what exercises booking end-to-end on a schedule and when it last ran. If the answer is “nothing does,” uptime is a floor and the critical path is still unmonitored.
The rest of the punch list
The false-green health check is the one that has to change before launch, because it’s actively lying. The rest is ranked so scope stays honest:
FIX THIS WEEK
After the health endpoint
- [MEDIUM] Monitor runs inside the deployment — move it to an external service pointed at the production URL.
- [MEDIUM] No synthetic check of the booking path — add the scheduled E2E or deep-check above.
- [MEDIUM] No certificate-expiry alert — one toggle on the external monitor; an expired cert is a silent total outage.
- [LOW] The alert channel is an email nobody watches. Route it to Slack/SMS so it interrupts, or it’s a chart, not a pager.
The shape: a monitor that lies is worse than no monitor, so it goes first; the rest is turning a green light into an actual safety net. Worst-first reads as “make the health check honest and the monitor external before launch, add the synthetic pass this week.”
ASK YOURSELF
“We already have monitoring — why is this a finding?” Because you have the shape of monitoring: an endpoint and a dashboard. What you don’t have is a check that can turn red. A health endpoint that always returns 200 and a monitor that dies with the app will both report calm through the exact outage you bought monitoring to catch. The fix isn’t more dashboards — it’s making the one you have capable of telling you the truth.
Why this keeps happening
Monitoring gets faked without anyone meaning to fake it. Adding a /health route that returns “ok” and pointing something at it feels like monitoring — the box gets ticked, the dashboard goes green, and green is exactly the outcome you wanted to see, so nobody interrogates it. The failure only reveals itself during a real outage, which by definition hasn’t happened yet on a pre-launch app. AI builders generate the endpoint and the config that produce a green light; producing a light that can go red at the right moment is a different, invisible kind of work.
That’s the gap launchworthy exists to close. It plays bouncer at the door of production: it knows a static-200 health check wired to a monitor is an active false green, that a checker living inside the deployment can’t report the outage that took it down, and that uptime green says nothing about whether signup still writes a row — so it refuses to mark monitoring as passing on a dashboard it can’t verify, and hands you the honest health endpoint, the external monitor, and the synthetic check instead.
The endpoint answered. The monitor was green. The box was ticked. And every one of them would have stayed calm while the database was on fire.