model rot
we run an outreach engine that one day started failing its model calls. a hardcoded model id had been retired upstream, the error was swallowed by a retry path, and the system kept "running" for sixteen days. nobody noticed, because every dashboard said fine. the thing that caught it was a flat cost ticker: spend dropped to zero, and zero on a system that should be spending is its own alarm.
two rules came out of that. first, model ids are config that rots. providers retire versions, aliases drift, and the failure is silent by default, so you pin the id and then canary it: a small scheduled call that proves the id still resolves, before your customers find out for you. second, watch the economics, not just the errors. error rates tell you when something breaks loudly. spend, volume and latency tell you when something breaks politely.
the helpful catch block
a catch block that synthesises green data is worse than an error. the page that renders "no data yet" instead of the real failure, the fallback that returns a plausible-looking default, the status page that computes uptime from the only checks it remembered to run. each one converts a loud, fixable failure into a quiet lie, and quiet lies compound.
the rule: fall back to nothing, or fall back loud. fabricated green is the one output a monitoring setup cannot recover from, because it poisons the very thing that would have told you. an honest red screen at 3am is a gift. a fake green one is an incident you get to discover twice.
the claim without the check
"done" is a verb with evidence. an agent that reports a deploy complete because the build passed is guessing: merged is not deployed, and the only acceptable proof is the live url answering correctly, checked, not assumed. we wire this in as a gate now. the word "done" does not pass unless a verify step has run against the real surface, and the gate refuses the claim otherwise.
the same rule applies to the humans reading the dashboards. never report a system's state from memory, or from the last time you happened to look. query the source of truth, every time, because state you didn't just check is a rumour. most "the monitoring lied to me" stories are really "the monitoring told the truth and nobody asked it" stories.
the pattern underneath
all three failures share a shape: the system kept talking after it stopped knowing. the fix is never a smarter model. it's a harness that treats silence, zeros and unverified claims as first-class signals, the same way it treats errors. build for the quiet failure and the loud ones take care of themselves.