§ CAPABILITY

QA and Test Strategy Consulting

A test suite people trust and run. Coverage placed where failure is expensive rather than spread evenly, and flakiness treated as a defect rather than as weather.

Who
Delivered by a senior team assembled for the engagement, against a defined scope.
Request a call How the work runs

What you're seeing

A red build gets re-run before anyone looks at it.
Usually means The suite has stopped gating anything. Every test still costs time to run and maintain, and none of them is preventing anything — this is the worst possible state, worse than having no tests.
Coverage is high and bugs still reach production.
Usually means Coverage was pursued as a number, so tests accumulated where they were cheap to write rather than where failure is expensive. The percentage is real and it is measuring the wrong thing.
The end-to-end suite takes forty minutes and half of it is testing form validation.
Usually means The pyramid is inverted — things testable in milliseconds underneath the interface are being tested through a browser. It is slow, brittle, and the brittleness is what teaches people to re-run.
Nobody can say what the suite is supposed to guarantee.
Usually means There is no written strategy, so what gets tested is decided per pull request by whoever is writing it. That produces uneven depth with no relationship to risk.

A suite nobody trusts is worse than none

Once a red build is routinely re-run before anyone reads it, the suite has stopped gating anything.

The cost has not gone anywhere. It still takes minutes on every push, engineers still maintain it, and it still blocks merges — it just does not prevent anything. That state is strictly worse than having no tests, because at least a team with no tests knows what it does not have.

Getting out of it is not primarily about writing more tests. It is about making every remaining failure meaningful, which usually means deleting some, quarantining the unreliable ones on a deadline, and moving assertions down out of the browser.

Coverage follows risk

Authorisation, money movement and data integrity get depth. A settings page that renders a form does not need the same treatment.

That sentence is the whole strategy, and the reason it needs writing down is that a coverage percentage says the opposite. A percentage is indifferent to what the code does — it rewards whatever is cheapest to reach, and the cheapest code to test is almost never the code whose failure ends up in a customer’s invoice.

The written strategy names both halves: what must not break, and what is deliberately uncovered. The second half is the part that makes it a decision rather than an aspiration, and it is the part a number cannot express.

Fast enough to block on

A suite that is too slow to gate a merge is a suite that runs after the decision it was supposed to inform.

So there is a runtime budget, enforced in CI like any other budget. Keeping inside it is what forces the pyramid to stay the right way up: most assertions at unit level where they cost milliseconds, a thin end-to-end layer over the flows that carry revenue, and integration coverage in between for the seams that genuinely need a running dependency.

Where it sits

This is a capability inside Software Development, where the strategy is written alongside the build rather than after it, and inside Platform Engineering, where the suite is one of the gates in the path to production.

It pairs with two others directly. Release & Deployment Verification covers what is checked after a deploy rather than before it — the two together decide what “safe to ship” actually means. And CI/CD Pipeline is where the runtime budget is enforced, because a strategy with no gate behind it is a document.

How the work runs

  1. Place coverage by risk

    Money paths, data integrity and authorisation get depth. Everything else gets enough. Uniform coverage targets produce tests where they are cheap rather than where they matter.

  2. Fix the pyramid

    Most assertions at unit level, a thin end-to-end layer over the flows that carry revenue. Inverted suites are slow, brittle, and eventually ignored.

  3. Kill flakiness

    Flaky tests quarantined and fixed on a clock. One unreliable test teaches the team to re-run red builds, which removes the value of every other test.

  4. Put it in the gate

    The suite wired into CI as a merge condition, with a runtime budget so it stays fast enough to actually block on.

What arrives

  • A written test strategy naming what is covered and what deliberately is not
  • A stable end-to-end suite over the critical flows
  • A flaky-test policy with quarantine and a repair deadline
  • CI gating with an enforced runtime budget

What it costs your team

Around three hours a week from a lead engineer, mostly reviewing what gets covered.

Next step

Tell us where this is breaking and we'll say what it takes to fix.

Request a call

How we decide

  • Coverage is placed by risk, and no percentage target is set

    Costs It removes the single number executives and tooling both like to track.

    A coverage percentage rewards testing whatever is cheapest to test, which is rarely where the money is. Authorisation, payment and data integrity get depth; a settings page that renders a form gets enough. The written strategy names what is deliberately not covered, which is the part a percentage can never express.

  • Flaky tests are quarantined and repaired on a clock, never tolerated

    Costs Quarantining removes real coverage temporarily, and the repair deadline creates work nobody planned for.

    One unreliable test teaches the team that red builds are re-run rather than investigated. That habit transfers instantly to every other test in the suite, including the ones that were about to catch something. Flakiness is not weather, it is a defect, and it is the defect with the largest blast radius.

  • The end-to-end layer stays thin, deliberately

    Costs It leaves some integration paths covered only indirectly, which feels risky when writing the strategy.

    End-to-end tests are the slowest and least stable form of coverage available. A thin layer over the flows that carry revenue is fast enough to gate a merge and stable enough to be believed. A thick one is neither, and a suite nobody believes is not coverage — it is a queue.

Frequently Asked Questions

A written statement of what the test suite is responsible for guaranteeing, at which level each guarantee is enforced, and what is deliberately left uncovered. Without one, those decisions get made per pull request by whoever is writing it, which produces uneven depth with no relationship to where failure is expensive.
None in particular, and setting one is usually counterproductive. A percentage rewards testing whatever is easiest to reach, and the easiest code to test is rarely the code whose failure costs money. We agree which paths must not break and cover those properly instead.
Usually not at this stage. Engineers testing their own work produces faster feedback than a separate team testing it afterwards, and a handover step adds latency to every change. What is needed is a strategy and someone who owns it — that is not necessarily a headcount.
Almost always because there are too many of them, testing through the interface things that could be tested underneath it. Every browser test is exposed to timing, network and rendering variability. A thin end-to-end layer over a solid unit layer is faster and considerably more stable.
No, and the confusion is worth clearing up. Exploratory testing finds things automation cannot, because automation only checks what someone already thought of. What should not be manual is regression — a human re-checking known behaviour before every release is the most expensive and least reliable way to do it.

Sources

Page reviewed