Skip to content
Leadership Advanced 4 min

Building a Blameless On-Call

The first question after an incident — 'who broke it?' or 'what broke?' — sets how candid the team will be from then on. Blamelessness is a precondition, not a reward.

By Victor Robin •

The problem

The first question an organisation asks after an incident — “who broke it?” or “what broke?” — is read instantly by everyone in the room, and they recalibrate how candid they’ll be from that moment on. Ask the first one and you forfeit the second one’s worth of information: the theories, the “here’s what confused me at the time,” the texture that makes a review useful.

I asked it. Forty minutes into a postmortem for a payments-queue stall, someone noted the failure started after the 14:02 deploy, and I said, “whose change was that?” Not unkindly — but I watched it land. The engineer who had shipped it, one of the most careful people on the team, answered and then gave me facts and nothing else for the rest of the hour. The bug was a misconfigured retry backoff, cheap to fix. What I’d done to the room was not.

The model

Operational excellence rests on trust and autonomy, and I’d had the order backwards. I thought blamelessness was a reward you give a mature team; it’s the precondition. You can’t ask people to own systems they’re afraid to be honest about. Blame doesn’t merely feel bad — it suppresses exactly the signal you need to get more reliable. The quiet engineer wasn’t sulking; she was being rational.

What to do

  • Move postmortems async and off the hot path. The first draft is written ~two days after the incident, in a shared doc, by whoever held the pager — not interrogated live while everyone is still raw.
  • Ban the name in the timeline. Timelines read “the 14:02 deploy” and “the retry config,” never “Maria’s change.” If a person genuinely matters, it’s because the system made it too easy for any reasonable person to make that mistake — and that is the finding.
  • Put a system between the alert and the human, so on-call opens a triage with the evidence already assembled: less staring at dashboards alone at 3am, less defensiveness, faster orientation.
  • Adopt SLOs and error budgets, so reliability is a target the whole team owns. A spent budget is a planning conversation, not a tribunal.

The results

Over two quarters the change I cared about most was qualitative: reviews got their texture back, and people started writing the “here’s what confused me” lines again — where the real fixes hide. The measurable parts, roughly: willingness to join the on-call rota stopped being a fight, moving from quietly losing people to a full, volunteered rotation; time-to-diagnosis came down by close to half once the agent assembled evidence first (a good chunk of that is better tooling, not pure culture); and it became a recruiting line that landed — more than one candidate said the way we ran incidents was why they signed.

The caveat: blamelessness can slide into no-accountability, and I’ve had to hold that line deliberately. Blameless means we don’t punish the human for the system’s gaps; it does not mean nobody owns the follow-up. The discipline is separating the two out loud — the system gets the scrutiny, a named person gets the action item — so “no blame” never becomes “no owner.”

Further reading