§ CAPABILITY

Observability Stack Implementation

Enough signal to answer a question you did not anticipate. Metrics, logs and traces joined by a common identifier, with the cost of collection under deliberate control.

Who
Delivered by a senior team assembled for the engagement, against a defined scope.
Request a call How the work runs

What you're seeing

An incident is investigated by reading logs in three tabs.
Usually means The signals are not correlated. Every investigation is three separate searches run under time pressure, and the join between them is happening in someone's head.
The observability bill grew and nobody changed a setting.
Usually means Cardinality. One unbounded label — a user id, a request id, a customer name attached to a metric — multiplies a time series into millions without anyone touching a configuration file.
Alerts fire on CPU and disk and nobody acts on them.
Usually means Alerting is on causes rather than symptoms. High CPU is not a problem; a failing checkout is. Cause-based alerts produce noise and train the rotation to dismiss.
Answering a new question about behaviour requires shipping code.
Usually means The system is monitored but not observable. Monitoring answers questions someone predicted; observability is being able to ask one nobody anticipated.

Three signals, one identifier

Metrics say something changed. Traces say where. Logs say what.

Without a shared identifier joining them, an incident becomes three separate investigations run in parallel under time pressure, with the join happening in the head of whoever is on call. That works when the person investigating built the system. It fails exactly when it matters most — an unfamiliar service, an unfamiliar failure, someone new on the rotation.

Propagating a trace identifier through logs and exposing it on metrics is not a large piece of work. It is the difference between a graph leading to a request and a request leading to a line of code, and the alternative to doing it is doing the join manually every time.

Alert on symptoms

An alert should mean a person must act now.

Cause-based alerting produces a rotation that dismisses pages, and it does that regardless of whether the underlying conditions are real. High CPU, elevated memory, a queue that is longer than usual — all are frequently normal, and all are frequently the leading edge of nothing at all.

Symptom-based alerting is narrower and it works: the checkout success rate has dropped, the API is returning errors to a customer, latency on the path that matters has crossed the threshold. Each of those has a runbook, because an alert whose response has to be invented at three in the morning is an alert that will be handled badly.

Cost is a design decision

Observability spend grows silently and arrives at a number nobody approved.

The mechanism is almost always cardinality. A label that looked useful — user id, request id, customer name — turns one time series into as many as there are distinct values, and the storage and query cost follows. No configuration changed; the traffic pattern did.

So sampling rates, retention tiers per signal and a cardinality cap are set deliberately at the start rather than after the first alarming invoice. This is also the point at which somebody has to decide what is worth keeping for a year and what is worth keeping for a week, which is a useful conversation independent of the bill.

Where it sits

This capability sits under Platform Engineering, where instrumentation is what makes everything else in the practice possible, and under Engineering Velocity, where it supplies the failure-rate and recovery-time halves of the delivery picture.

It is the foundation two neighbours stand on. On-Call & Incident Response is a rotation of people being woken with no information unless this exists first. And SRE & SLO Design requires a measured indicator before a target means anything — you cannot set an objective for something you are not measuring where the user is.

How the work runs

  1. Instrument the paths that carry the business

    Starting from the flows that matter commercially rather than from what the agent collects by default.

  2. Correlate the three signals

    A trace identifier through logs and metrics so a graph leads to a request and a request leads to a line of code, without three separate searches.

  3. Alert on symptoms, not causes

    Pages tied to what a user experiences. Alerting on CPU produces noise; alerting on the checkout failing produces action.

  4. Control the cost

    Sampling, retention tiers and cardinality limits set deliberately, because observability spend grows silently and reaches a number nobody approved.

What arrives

  • OpenTelemetry instrumentation across the critical paths
  • Dashboards for on-call and a separate set for service owners
  • Alert rules tied to user-visible symptoms, each with a runbook link
  • A cost model with sampling and retention set per signal

What it costs your team

Around four hours a week from a platform engineer during instrumentation.

Next step

Tell us where this is breaking and we'll say what it takes to fix.

Request a call

How we decide

  • Instrumentation starts from the paths that carry the business

    Costs It leaves parts of the system less instrumented than a uniform rollout would.

    Default agent collection produces enormous volume weighted toward whatever is easiest to collect, which is rarely what a commercial incident is about. Starting from sign-up, checkout and the API enterprise customers depend on means the first useful investigation happens sooner and the bill stays proportionate to the value.

  • OpenTelemetry regardless of the backend

    Costs It is slightly more setup than a vendor's own agent, which is designed to be effortless.

    The instrumentation is the expensive, long-lived asset and the vendor is the reversible decision. Instrumenting with a proprietary agent inverts that — it makes the cheap decision permanent, and re-instrumenting a whole estate to change vendor is the reason organisations stay on a bill they have outgrown.

  • Alerts fire on symptoms, and every one has a runbook

    Costs It means deleting alerts that fire on genuinely real conditions.

    A page should mean a person must act now. Anything else belongs on a dashboard and is looked at deliberately. An alert with no runbook is an alert whose response is being invented at three in the morning by whoever is least equipped to invent it, and an alert nobody acts on trains the rotation to dismiss the one that mattered.

Frequently Asked Questions

Being able to answer questions about a system's behaviour without shipping new code to answer them. Monitoring tells you that a known thing broke; observability lets you investigate something nobody predicted. The practical difference is whether an unfamiliar incident is a debugging session or a code deploy.
Metrics are aggregates over time — they tell you something changed. Traces follow one request through the system — they tell you where. Logs are the detail at a point — they tell you what. Each is weak alone, and the value comes from a shared identifier that lets one lead to the next.
Instrument with OpenTelemetry regardless, so the vendor decision stays reversible. After that, Grafana with Prometheus if you are willing to run it, or a hosted platform if you would rather not. The difference is operational cost rather than capability, and the choice should be made on which cost you prefer to pay.
Because they are on causes rather than symptoms. High CPU is not a problem in itself — plenty of healthy systems run hot. A page should mean a person needs to act now, and everything else belongs on a dashboard that gets read deliberately rather than at three in the morning.
Sampling on high-volume traces, retention tiers by signal, and a hard cap on metric cardinality. Cardinality is the usual culprit: attaching a user id or a request id as a metric label turns one series into millions, and the bill moves without anybody changing a setting.

Sources

Page reviewed