§ WHAT WE FIX

AI Consulting for Engineering Teams

Two problems that look separate and are not: your team has AI tooling and no measurable return from it, and your product has an AI feature that costs more than it earns. Both are engineering problems.

Who
Founder holds the engineering leadership seat; the delivery team executes underneath it.
Request a call What this covers

What you're seeing

Licences were bought for the whole team and nobody can say what changed.
Usually means Adoption was measured by seats rather than by delivery. Seat counts are always high, because the licence is free to the engineer and awkward to decline.
The feature is impressive in a demo and unpredictable in production.
Usually means There is no evaluation set, so every change to a prompt or a model is an argument rather than a measurement. It is the single most common reason an AI feature stalls after launch.
Inference spend is growing faster than the usage behind it.
Usually means Context is being sent that nobody needs, the same request is being paid for repeatedly, or a frontier model is answering questions a smaller one would answer identically. All three are design decisions, not vendor pricing.
Legal or procurement has asked which of your data leaves your infrastructure and nobody has the answer written down.
Usually means The data boundary was decided implicitly by whoever picked the provider. It is far cheaper to answer at design time than during a security review with a deal attached to it.
You are being asked to produce an audit trail and you have model traces.
Usually means Those are different records. A trace says what was generated; an audit log says what was done and on whose authority. Teams usually discover the gap during fieldwork, which is the worst moment to discover it.

Two problems, one page

The boundary between “our team gets nothing from these tools” and “build AI into our product” runs inside a single engagement rather than between two competing pages.

The same three questions decide both. What is being measured. What it costs per unit of work. What happens when the output is wrong. A team rolling out coding assistants without a delivery baseline and a team shipping an LLM feature without an evaluation set have made the identical mistake, and it produces the identical symptom — nobody can tell whether it is working, so nobody can improve it and nobody can stop.

Evaluation is the asset

The durable output of this work is not a prompt and not a model choice. It is a graded set of cases with expected behaviour written down.

With one, a model change is a measurement that takes an afternoon. A prompt edit either moves the number or it does not. A cheaper model either passes or it does not, and the cost decision stops being a debate about quality anybody can hold an opinion about. Without one, every change is argued from anecdote, and the anecdotes are selected by whoever is most invested in the outcome.

It also makes stopping available. A feature nobody can grade tends to stay shipped, because removing it feels like an admission rather than a decision. A graded one can be retired on evidence.

Building the set is the unglamorous part: sitting with the people who know what a good answer looks like and getting it written down, including the cases where the correct behaviour is to refuse. It usually takes days, not weeks, and almost nobody does it before shipping.

Where the money goes

Inference spend grows quietly because nothing else in a normal stack is priced per request the way a model is. There is no connection pool to exhaust and no server to fall over — the bill just moves.

Three things account for most of it. Context sent that nobody needs, usually because the retrieval step returns ten documents when two would do. The same answer bought repeatedly, because nothing caches at the semantic level. And a frontier model answering questions a smaller model answers identically, which is only visible once you have something to compare against.

All three are design decisions, and all three are much cheaper to make before the feature has traffic than after.

The data boundary, before the provider

The question that stalls AI features late is rarely technical. It is a security review, from a customer or an auditor, asking which of their data left your infrastructure and what the provider retained.

Answering that after the fact means either re-architecting or explaining why you cannot answer. Answering it first constrains the model choice, and that constraint is worth accepting: it is the difference between an AI feature you can sell to an enterprise buyer and one you have to carve out of the contract.

Where a certification is already in play, this work overlaps with SOC 2 Readiness directly — the model provider is a subprocessor, and it lands in vendor review like any other. Where the feature is a substantial piece of product rather than an addition to an existing one, it is built under Software Development with the evaluation and cost work folded into the same engagement. The internal half of this page — getting a measured return out of AI coding tools inside the engineering team rather than out of the product — is scoped on its own as AI Coding Tool Rollout and Enablement.

What this covers

Each of these is a capability with its own page, its own order of work and its own outputs.

Capability What it means
AI Engineering Enablement Getting a measurable return from AI coding tools. Baseline first, then rollout with the review and security practices adjusted for code a human did not write line by line.
AI App Development LLM features that hold up in production. An evaluation set before the feature ships, cost controlled by design, and data boundaries settled before a provider is chosen.

Next step

Describe the situation and we'll tell you whether this is the right shape.

Request a call

How we decide

  • The evaluation set is built before the feature ships

    Costs It delays the first release, and building it means writing down what a good answer is, which is harder than it sounds.

    A feature nobody can grade is a feature nobody can improve, and the argument about whether a prompt change helped will otherwise be settled by whoever is most senior. It also makes stopping available: without a grade, a mediocre feature stays shipped because removing it feels like an admission.

  • The model choice is treated as reversible and the harness as permanent

    Costs It costs work up front to route through an abstraction nobody needs on day one.

    The best model for a given call path changes every few months, and the cost curve changes faster. What survives is the evaluation harness and the routing, so that is where the effort goes. Committing the codebase to one provider's SDK converts a pricing decision into a refactor.

  • Data boundaries are decided before a provider is chosen

    Costs It can rule out the model you wanted before anyone has tried it.

    Which fields may leave your infrastructure, what a provider retains, and whether an EU-resident or self-hosted deployment is required are answers procurement and your auditor will need. Retrofitting them means either re-architecting or explaining to a customer why you cannot answer.

  • Tooling rollout is measured against delivery metrics, not surveys

    Costs It is unflattering — the honest number is usually smaller than the one people report.

    Self-reported time savings are the metric every vendor quotes and they measure enthusiasm. Cycle time, review latency and change failure rate were already being collected for other reasons, they have a pre-rollout baseline, and they cannot be improved by anyone feeling good about a tool.

Four ways to put a model behind a feature

The choice is usually presented as a quality question. It is mostly a cost and data-boundary question, and the quality difference narrows every quarter.

Approach Cost shape Data boundary When it wins
Frontier hosted API Per token, no floor, scales with traffic Data leaves your infrastructure under the provider's terms Reasoning-heavy paths, and anything still changing weekly
Smaller hosted model Per token at a fraction of the rate Same as above, different provider terms High-volume classification and extraction, once an evaluation set proves it holds
Self-hosted open weights Fixed infrastructure cost, indifferent to volume Nothing leaves; you own retention and residency Regulated data, or volume high enough that the fixed cost undercuts per-token
Fine-tuning Training cost up front, plus inference, plus a rerun on each base-model change Depends on where it runs A narrow, stable task where prompting has measurably plateaued

Most teams reach for the fourth row and need the second. Fine-tuning is the answer to a problem you can only recognise once an evaluation set shows prompting has stopped improving — which is another reason the harness comes first.

Is this you?

  • Licences were bought, adoption is uneven, and nobody can say what changed
  • An LLM feature works in a demo and behaves unpredictably in production
  • Inference spend is growing faster than the usage that justifies it

How we run it

Evaluation before rollout

A feature nobody can grade is a feature nobody can improve. We build the evaluation set before the feature ships, so a change to a prompt or a model is a measurement rather than a debate — and so the decision to stop is available.

Frequently Asked Questions

Building and running the parts of a product that call a model, with the same discipline applied to everything else that reaches production: measurement, cost control, failure handling and a data boundary you can describe. It also covers getting a return from AI tools inside the engineering team itself. The unusual part is the failure mode — a system that is confidently wrong rather than unavailable — and most of the practice exists to handle that.
Two things that look separate and are not. Getting a measurable return from AI coding tools inside your own team, and building model-backed features into your product so they hold up under real traffic. The same three questions decide both: what is measured, what it costs per unit of work, and what happens when the output is wrong.
Against the delivery metrics used everywhere else — cycle time, review latency, change failure rate — with a baseline taken before rollout. Self-reported time savings are not evidence, and they are almost always what gets reported. Where the honest answer is that nothing moved, that is a finding worth having before the next licence renewal.
Whichever one your evaluation set says is good enough at the lowest cost, and that answer changes every few months. The durable asset is the evaluation harness and the routing around it, not the choice itself, which is why we build the harness first and treat the provider as replaceable.
Data boundaries get decided before a provider is chosen, not after. Which fields can leave your infrastructure, what is retained and for how long, and what a self-hosted or EU-resident deployment would cost. These are the questions procurement and your auditor will ask, and they are far cheaper to answer at design time than during a security review.
No. Model traces record what a system generated; an audit log records what it did and on whose authority. They are different records with different retention requirements, and conflating them is how a team discovers during fieldwork that it cannot reconstruct who authorised an action.
Three levers, applied in order: send less context, pay for the same answer once by caching, and route each call path to the cheapest model that passes its evaluation. All three are design decisions rather than negotiations with a vendor, and all three are cheaper to make before the feature has traffic.
For building features on top of hosted models, usually not — it is backend engineering with an evaluation discipline attached. You need one when you are training or fine-tuning your own models, serving them yourself at scale, or working on a research problem rather than a product one.

Sources

Page reviewed