§ BUILD & DELIVERY

Platform Engineering and SRE

The system stays up because people are watching it, and that is not a strategy. We build the pipeline, the observability and the on-call practice that make reliability a property of the platform rather than of who is awake.

Who
Delivered by a senior team assembled for the engagement, against a defined scope.

What you're seeing

Incidents are found by customers before they are found by monitoring.
Usually means You have metrics but no alerts on the paths that carry the business, which is a different thing. Dashboards are for investigating; alerts are for being told.
Deploys need a person to shepherd them.
Usually means Release is a procedure rather than a property of the pipeline. It works until the person who knows the procedure is on holiday, and it silently caps how often anyone is willing to ship.
The same two engineers are woken up for everything.
Usually means Escalation follows knowledge rather than a rota, because the runbooks are in their heads. It is a retention risk before it is a reliability one.
An enterprise contract has arrived with an availability commitment in it.
Usually means You now owe a number to a customer and probably cannot currently measure it. Defining the SLI is the work; the SLO is the easy part after that.
Post-incident reviews produce a document and no change.
Usually means The review has become an obligation rather than a mechanism. A review that does not end in owned, dated work is a record of an incident rather than a defence against the next one.
Nobody can say what a deploy costs in risk, so everything is batched.
Usually means Change failure rate is unmeasured, so caution substitutes for evidence. Batching then raises the actual risk of every release, which confirms the caution.

Reliability as a property, not a rota

Systems that stay up because a particular person notices things are fragile in a way that no dashboard reports.

It is not usually visible as risk from inside. Availability looks fine. What is actually happening is that a small number of people are carrying the difference between the system and a reliable system, in their attention and in what they know that is not written down. That works, and it keeps working, right up until one of them is on a plane.

The work here moves that dependency into the platform: alerts that mean something, a deployment path boring enough that nobody supervises it, runbooks written by the people who currently do not need them, and an incident practice that changes the system afterwards rather than producing a document.

Alerts, and why fewer is the goal

The most common thing we delete in the first month is alerts.

An alert that does not require action, at the moment it fires, teaches the rotation that alerts do not require action — and it teaches that regardless of whether the underlying condition is real. The cost is invisible until the night something genuinely broke and the page was dismissed with the others.

So the set is derived from the paths that carry the business rather than from what the tooling makes easy to graph, and each surviving alert has a written reason and a runbook. Ten alerts nobody dismisses beat a hundred that are mostly noise. This is unglamorous work, it is largely subtraction, and it does more for a rotation’s quality of life than any tool purchase.

The error budget is the decision

An SLO that never changes what gets built is decoration.

What makes it real is a sentence agreed in advance: when the budget for the period is spent, feature work pauses and reliability work takes priority until it recovers. That sentence has to exist before an incident, because it is impossible to negotiate credibly during one — everyone involved has an interest and the pressure is asymmetric.

Which means the target is set alongside the people who own the roadmap, not inside engineering. That conversation is the substance of the work; the arithmetic of the budget is trivial by comparison.

Where it overlaps compliance

Change management, access review and incident response appear in a SOC 2 control set and in any serious reliability practice. Built once, they serve both, and the evidence collects itself as a side effect of doing the thing properly.

Where a certification is already in play, that overlap is the cheapest part of the programme and worth sequencing deliberately — SOC 2 Readiness shares this page’s approach to controls, and running the two together avoids building each of them twice in slightly incompatible ways.

Where the symptom is that delivery is slow rather than that the system is fragile, the diagnosis belongs to Engineering Velocity; the two share four capabilities and the distinction is which constraint is binding. And where this work is part of building the product itself rather than stabilising one that exists, it runs inside Software Development.

What this covers

Each of these is a capability with its own page, its own order of work and its own outputs.

Capability What it means
DevOps Consulting Infrastructure described in code, environments that match each other, and a deployment path that does not depend on knowing who set it up.
CI/CD Pipeline A pipeline fast enough that people wait for it. Build time is a delivery constraint, and a twenty-minute pipeline changes how a team batches work whether anyone intends it to or not.
Observability Stack Enough signal to answer a question you did not anticipate. Metrics, logs and traces joined by a common identifier, with the cost of collection under deliberate control.
On-Call & Incident Response A rotation people can sustain and a response practice that changes the system afterwards. Who gets paged, for what, and what happens the following week.
SRE & SLO Design Reliability targets that change decisions. Objectives set against what users actually experience, with a written agreement about what happens when the budget is spent.
Developer Experience The friction between an engineer having an idea and the change being live. Local setup, build times, environment access — measured, then removed in the order that costs the most.
Release & Deployment Verification Deploying stops being an event. Changes go out behind flags, in stages, with the verification automated and a rollback path that has been tested rather than assumed.
QA and Test Strategy A test suite people trust and run. Coverage placed where failure is expensive rather than spread evenly, and flakiness treated as a defect rather than as weather.

How we decide

  • The paved path is built before a platform team is formed

    Costs It leaves platform work distributed for longer than an org chart would like.

    A platform team chartered before there is a platform produces process — standards, review boards, requests — because that is what is available to produce. Build the path to production first and let its maintenance grow into a role, and the team arrives with users it already serves.

  • Alerts are derived from the paths that carry the business

    Costs It means deleting alerts that fire on real conditions, which is uncomfortable to argue for.

    An alert that does not require action teaches the rotation to ignore alerts, and it does that whether or not the underlying condition matters. Ten alerts nobody dismisses are worth more than a hundred that are mostly noise, and the honest cost of the hundred is paid the night one of them was real.

  • The error budget policy is agreed before an incident, in writing

    Costs It commits the roadmap in advance to something nobody wants to commit to.

    An SLO that never changes what gets built is decoration. What makes it a decision is the sentence saying what happens when the budget is spent — feature work stops, reliability work takes priority — and that sentence is impossible to negotiate credibly during the incident that spends it.

  • Reliability controls are built once for reliability and for the audit

    Costs It slows the reliability work down to accommodate evidence requirements that are not otherwise urgent.

    Change management, access review and incident response appear in a SOC 2 control set and in any serious reliability practice. Building them twice is common, expensive, and produces two versions that disagree — which is itself an audit finding.

Where reliability work can sit

Four arrangements for the same problem. They differ in what is left behind, which is the thing to compare.

Arrangement What it produces What is left behind When it fits
Platform and reliability engagement Pipeline, observability, on-call practice and SLOs, built with your engineers A paved path your team owns and can change Reliability is a constraint on the business and there is nobody to own it yet
Buying an observability tool Telemetry, and a bill that grows with it Data, and the same unanswered question about what to alert on You already know what matters and only lack instrumentation
Managed SRE or NOC service Somebody watching, on a contract A dependency, and an escalation path to your team anyway 24/7 coverage is required now and cannot be staffed
Hiring an SRE Everything, eventually, in their order A person, and whatever they built while they were there You can wait two quarters and have work to keep them busy after

The second row is the most frequent purchase and it answers a question nobody asked. Instrumentation is rarely the constraint; deciding what is worth waking someone for is, and no vendor can decide that for you.

Is this you?

  • Incidents are found by customers before they are found by monitoring
  • Deploys need a person to shepherd them
  • An enterprise contract has arrived with an availability commitment attached

How we run it

An error budget is a decision, not a dashboard

An SLO that never changes what gets built is decoration. We set targets alongside the people who own the roadmap, and write down in advance what happens when the budget is spent — because that agreement, made before an incident, is the only part that survives one.

Frequently Asked Questions

Building the internal surface your product engineers work against — the pipeline, the environments, the observability, the paved path to production — so that shipping does not require knowing how the infrastructure is assembled. The useful framing is that it is a product with internal users, which means it has adoption, a feedback loop, and the possibility of being ignored.
Platform engineering builds the path; SRE is the practice of running things on it to a stated reliability target. They overlap heavily in a company of this size and are often the same people. The distinction that matters operationally is that one is measured by developer adoption and the other by error budget.
No, and starting with the team is usually the wrong order. A platform team formed before there is a platform produces process, because that is what is available to produce. Build the paved path first and let its maintenance grow into a role.
From the paths that carry the business rather than from what the tool makes easy to graph. Sign-up, checkout, the API call your enterprise customers depend on. A dashboard of everything is a dashboard nobody reads at three in the morning, and an alert nobody acts on trains the rotation to ignore the one that mattered.
Rotation and compensation, what pages a human versus what waits until morning, the escalation path, runbooks for the things that actually page, and a review practice that produces owned work rather than a document. The alert routing is the easy half. The agreement about what is worth waking someone for is the rest of it.
A service level objective is a target for a measured user-facing signal — say, the share of requests served successfully within a latency bound over a rolling month. You need one when reliability has to be traded against feature work explicitly, which is usually when a customer contract puts a number on it. Before that, a small number of good alerts is worth more.
It usually should. Change management, access review and incident response are controls a SOC 2 auditor will ask about and practices any reliable system needs. Building them once for both purposes is considerably cheaper than building them twice, and two versions that disagree is itself a finding.
All of it, and that is the design constraint rather than an aspiration. The pipeline is code in your repository, the alerts have stated reasons, and the on-call practice is run by your engineers from the first rotation with us alongside rather than instead.

Sources

Page reviewed