§ CAPABILITY

On-Call and Incident Response Design

A rotation people can sustain and a response practice that changes the system afterwards. Who gets paged, for what, and what happens the following week.

Who
Founder holds the engineering leadership seat; the delivery team executes underneath it.
Request a call How the work runs

What you're seeing

The same two engineers are woken for everything.
Usually means Escalation follows knowledge rather than a rota, because the runbooks are in their heads. It is a retention risk before it is a reliability one, and it usually resolves by one of them leaving.
The pager is on the founder's phone.
Usually means There is no rotation and a single point of escalation who also cannot be replaced during an incident. It is among the cheapest things on any reliability list to fix, and reliably the last to get attention.
Post-incident reviews produce a document and the same failure recurs.
Usually means The review has become an obligation rather than a mechanism. Actions without owners and dates are a record of an incident, not a defence against the next one.
Nobody declares incidents because declaring one feels like an accusation.
Usually means The severity definitions are missing or the culture around them is punitive. Undeclared incidents are handled slower, by fewer people, with no record — which is strictly worse in every dimension.

Decide in daylight

What is worth waking someone for is a judgement best made calmly, written down, and then simply followed at three in the morning.

The alternative is that the decision is made under load by whoever is on the rotation, with incomplete information and a strong incentive to escalate rather than be wrong. That produces two failure modes at once: pages that did not need a human, and hesitation on the ones that did.

So the paging policy is written in advance and it is short. What wakes someone. What waits for morning. What goes to a dashboard and is never a page at all. Every entry in the first category has a runbook, because an alert whose response has to be invented in the moment is an alert that will be handled badly.

Anyone can declare

The right to declare an incident should not depend on seniority.

Where it does, incidents get declared late — at the point where they are undeniable, which is also the point where the cheap responses have expired. The engineer who noticed something at ten past two waited for someone senior to agree, and the twenty minutes that cost is the difference between degraded and down.

A declaration that turns out to be nothing costs half an hour and a slightly awkward message. The asymmetry is not close, and the policy should say so explicitly rather than leaving people to infer it.

Review that changes something

The test of an incident practice is whether the same failure recurs.

If reviews produce documents rather than owned actions with dates, it will. That is not a failure of diligence — the document is usually thorough — it is that nothing in the format connects to the backlog, so the actions are aspirational and compete with a roadmap that has owners.

Blameless is the other half. Asking what about the system permitted an action to cause harm, rather than who performed it, is not softness. It is the only version that keeps near-misses being reported, and near-misses are where the information is cheapest to act on.

Where it sits

This capability sits under Platform Engineering, where the rotation and the review practice are what turn instrumentation into a system that improves.

It depends on Observability Stack — a rotation without instrumentation is a schedule of people being woken with no information — and it pairs with SRE & SLO Design, which supplies the severity thresholds and the error budget that an incident actually spends.

How the work runs

  1. Design the rotation

    Size, frequency, handover and compensation. A rotation that burns people out is a retention problem disguised as a reliability practice.

  2. Set the paging policy

    What wakes someone and what waits for morning, agreed in daylight rather than decided during an incident.

  3. Define severity and escalation

    Levels with concrete criteria, a named escalation path, and who is authorised to declare an incident — which should be anyone.

  4. Install blameless review

    A review format that produces owned actions with dates. A postmortem that ends in a document is a document.

What arrives

  • A staffed rotation with handover and compensation agreed
  • A paging policy separating what wakes someone from what waits
  • Severity definitions with escalation paths and runbook links
  • A review format with an action tracker rather than an archive

What it costs your team

A design workshop and about two hours a week from engineering leadership during setup.

Next step

Tell us where this is breaking and we'll say what it takes to fix.

Request a call

How we decide

  • The rotation is sized so nobody carries it more than one week in four

    Costs It requires enough participants, which sometimes means widening who is eligible before the rotation can start.

    Below that ratio on-call stops being a duty and becomes an attrition mechanism, and it selects against exactly the senior people who are best at it. A rotation that burns people out is a retention problem wearing a reliability practice's clothes, and the cost surfaces two quarters later as a resignation.

  • On-call is compensated

    Costs It is a real cost that appears in a budget where it previously did not.

    Being unable to leave the house, drink, or be unavailable for a week is work whether or not the phone rings. Treating it as included in salary is the fastest way to lose the people who are good at it, and the replacement cost of one senior engineer exceeds a year of the allowance.

  • Anyone may declare an incident

    Costs It produces some declarations that turn out to be nothing.

    The alternative — declaration requiring seniority or permission — means incidents get declared late, when they are undeniable and the cheap responses have expired. A false declaration costs half an hour. A late one costs the difference between a degradation and an outage, and the asymmetry is not close.

  • Reviews end in owned, dated actions or they have not happened

    Costs It generates work that competes with the roadmap, immediately after an incident has already cost time.

    The test of an incident practice is whether the same failure recurs. A postmortem that ends in a document is a document, and the organisation learns that reviews are a ritual to be completed. Actions with a named owner and a date are the only part of the format that changes the system.

Frequently Asked Questions

The practice around unplanned service disruption: how it is detected, who is notified, how severity is assessed, who coordinates, how it is communicated, and what happens afterwards. The engineering to fix the problem is the part everybody focuses on; almost all of the improvable variance is in the surrounding process.
Enough that nobody carries it more than one week in four, and enough that a single absence does not break the schedule. In practice that is six people or more. Below that, on-call becomes an attrition mechanism and it selects against the senior engineers you most want to keep.
Generally yes — it is the shortest feedback loop between a design decision and its operational consequence, and it changes design behaviour faster than any review process. It has to come with the authority to fix what pages them, though. Without that it is an obligation with no lever, which is the version people resent.
Yes. Being tied to a laptop and unable to be unavailable is work regardless of whether the phone rings. Organisations that treat it as included in salary lose the people who are good at it, and that is a considerably more expensive outcome than the allowance.
Asking what about the system allowed an action to cause harm, rather than who took the action. The second question is not more rigorous — it is less, because it stops people reporting near-misses and near-misses are where most of the useful information is.

Sources

Page reviewed