BusinessFinding production bugs before your customers do
A 15-person team cut serious production incidents by 80% with a rotating duty slot before daily. No new tooling — just someone whose job it was to look.
There is a specific kind of bad morning in engineering: support forwards a customer complaint, you go looking, and you find the problem started days ago. The bug isn’t the expensive part. The delay is.
An iGaming startup with a 15-person dev team had exactly this pattern. Production problems surfaced when customers or support escalated them — which meant, by definition, after the problem had grown large enough for someone outside the company to notice.
The fix was organisational, not technical
The change was a rotating devops duty. Each day, one person from the team — rotating, so it was never anyone’s permanent job — checked logs and metrics before daily standup and was first responder for production errors that day.
That’s it. No new observability platform. No alerting overhaul.
The result: serious production bugs dropped by 80%.
Why it works better than it sounds
It’s tempting to read that as “they started looking at dashboards.” The mechanism is more specific than that.
It assigns the responsibility to a person, not to the team. Monitoring that is everyone’s job is nobody’s job. Dashboards that exist but have no named owner get glanced at during incidents and ignored otherwise. Naming one person per day converts a vague good intention into a task with an owner.
It attaches the check to something that already happens. Daily standup is a fixed point in the day that nobody skips. Hanging the duty off it means the check inherits that reliability instead of needing its own discipline.
Rotation spreads the knowledge. When one person owns monitoring permanently, they become the only one who can read the system’s normal. Rotating means the whole team learns what healthy looks like — which is exactly the knowledge you need to notice unhealthy quickly.
It shifts detection earlier, and cost scales with delay. A problem caught in the first hours is usually a small fix. The same problem caught after a customer reports it has usually spread, generated support load, and damaged trust. The lesson from that engagement was blunt: a problem detected early costs a fraction of a problem reported by a customer.
The failure mode this catches
Consider a related incident, in a different iGaming company with a 10-person team.
A developer shipped a database change that had only ever been tested against a small dataset. Code review passed — the change looked correct, and it was correct, at small scale. On production it hit real data volumes and took down the entire database and the service on top of it for an hour.
The remedy was a mandatory procedure: migrations touching critical tables must be checked against a load-test or dev environment with production-scale data before they ship — not code review alone.
These two stories are the same story from different ends. In both cases the technical failure was ordinary. What made it expensive was that nothing was watching at the point where the failure would first show, and code review can’t watch for a class of problem that only appears at scale.
What to steal from this
If your team finds out about production problems from other people, three questions are worth asking before you buy anything:
- Does someone have the explicit job of looking today? Not “we have dashboards” — is there a name attached to today’s date?
- Is that check attached to a fixed point in the day that already happens reliably?
- Do your pre-production checks exercise production-scale conditions, or only correctness at small scale?
None of these require a purchase. They are all scheduling and process decisions, which is usually why they don’t get made — there’s no vendor to prompt you.
The uncomfortable general lesson
Both stories point at the same thing, and it applies well beyond production monitoring:
A system that nobody is scheduled to look at is not being monitored, no matter how good the instrumentation is.
The instrumentation is necessary. It’s not sufficient. The 80% reduction didn’t come from better data — the logs and metrics already existed. It came from someone being responsible for reading them at a predictable time, before anyone outside the company had a chance to notice first.
If delivery in your team has slowed and nobody can point to why, that’s the shape of problem a dev team performance audit is built to find.