§ CAPABILITY

AI App Development: LLM Features in Product

LLM features that hold up in production. An evaluation set before the feature ships, cost controlled by design, and data boundaries settled before a provider is chosen.

Who
Delivered by a senior team assembled for the engagement, against a defined scope.
Request a call How the work runs

What you're seeing

The feature demos well and behaves unpredictably with real users.
Usually means It was built against a handful of inputs somebody chose. Real traffic contains the phrasings, the empty fields and the adversarial cases nobody thought to try, and there is no harness to catch the difference.
Somebody changed a prompt and nobody can say whether it improved anything.
Usually means There is no evaluation set, so quality is now a matter of opinion and it drifts silently. Every subsequent change compounds the uncertainty.
Inference cost per active user is rising and nobody planned it.
Usually means Context size, model tier and caching were never decided per call path. All three are design choices, and they are considerably cheaper to make before the feature has traffic.
A customer asked what happens to their data and the answer took a week.
Usually means The data boundary was set implicitly by whoever chose the provider. It is a design decision that was made by default, and it is now a sales blocker.

Evaluation before the feature

An AI feature nobody can grade cannot be improved and cannot be safely changed.

The harness is what converts a model swap from a gamble into an afternoon’s measurement. It is what lets you take the cheaper model when the cheaper model is good enough, and it is what tells you when it is not. It is also what makes retirement possible — an ungraded feature stays shipped indefinitely, because removing it feels like admitting a mistake rather than reading a number.

Building it means sitting with the people who know what a good answer looks like and writing that down, including the cases where the correct behaviour is to decline. That is a day of domain experts’ time and it is not delegable to engineers, which is the main reason it gets skipped.

Cost is a design output

Inference spend grows quietly, because nothing else in a normal stack is priced per request the way a model is. There is no pool to exhaust and no server to fall over. The bill simply moves.

Three levers account for most of it. Context that is larger than the task requires, usually because retrieval returns ten documents when two would do. The same answer purchased repeatedly, because nothing caches at the level where caching would help. And a frontier model answering questions a smaller model answers identically, which is invisible until there is an evaluation set to compare against.

All three are architectural, and all three are far cheaper to decide before the feature has users.

Traces are not audit logs

A model trace records what was generated. An audit log records what the system did and on whose authority.

They have different retention requirements, different access controls and different readers, and conflating them is discovered during an audit — at the point where somebody asks who authorised an action and the only available record is a prompt and a completion. Where a feature takes consequential action on a user’s behalf, the two records are built separately from the start.

Where it sits

This is a capability under AI Consulting, which also carries the internal half — getting a measurable return from AI tooling inside the engineering team — and under Software Development when the feature is part of a larger build.

It leans on two neighbours. Backend & API supplies the retrieval layer, the caching and the interfaces the feature is reached through. And Observability Stack is where per-call cost, latency and failure rate actually become visible — an AI feature with no instrumentation is a bill with no explanation attached.

How the work runs

  1. Build the evaluation set first

    Real inputs with graded outputs, before the feature. It is what turns a prompt change from an argument into a measurement.

  2. Design the retrieval

    Where grounding is needed: chunking, embedding and ranking tested against the evaluation set rather than assumed from a tutorial.

  3. Control cost per call path

    Model tier, context size and caching decided per path. Inference cost is a design output and it is expensive to retrofit.

  4. Settle the data boundaries

    Which fields leave your infrastructure, what is retained, and what a self-hosted or EU-resident option would cost. Procurement and your auditor will both ask.

What arrives

  • An evaluation harness with graded cases, running in CI
  • A retrieval pipeline tuned against measured results
  • A per-path cost model with caching and model tier chosen
  • A data-flow record naming what leaves your infrastructure

What it costs your team

Domain experts to grade the evaluation set — usually a day in total, and not delegable to engineers.

Next step

Tell us where this is breaking and we'll say what it takes to fix.

Request a call

How we decide

  • The evaluation set is built before the feature ships

    Costs It delays the first release and requires a day of domain experts' time that cannot be delegated to engineers.

    A feature nobody can grade cannot be improved and cannot safely be changed. With a harness, swapping a model is an afternoon's measurement; without one it is a gamble taken on the strength of a demo. It also makes stopping possible — an ungraded feature stays shipped because removing it feels like an admission.

  • Cost is designed per call path, not negotiated with a vendor

    Costs It adds routing complexity early, for a bill that is currently small.

    Three levers account for most inference spend: how much context is sent, whether the same answer is paid for twice, and whether a frontier model is answering a question a cheaper one answers identically. All three are architectural. Retrofitting them into a live feature means changing the shape of every call while it is serving users.

  • There is an explicit path for the model being wrong

    Costs It constrains the product design, usually by requiring a human step somewhere the demo did not have one.

    The failure mode of these systems is confident incorrectness, not unavailability, and a feature with no wrong-answer path will find its own — in front of a customer, at the point where the output has already been acted on. Confidence thresholds, a review step for consequential actions, and a record of what was generated are the difference between a bad answer and an incident.

Frequently Asked Questions

Building a product feature on top of a language model: designing the context it receives, retrieving the grounding data it needs, evaluating whether its outputs are good enough, controlling what each call costs, and handling the cases where it is wrong. The API call itself is the smallest part of the work.
A set of real inputs with expected outputs, graded by someone who knows the domain, run automatically against every change. Without it, every prompt edit is a matter of opinion and quality drifts in whichever direction the last change happened to push it.
Only if you need retrieval, and often pgvector inside the Postgres you already run is sufficient well past the point people expect. A separate vector store is another system to operate, secure and back up, for a benefit most products never reach.
Retrieval-augmented generation — fetching relevant documents and putting them in the model's context so answers are grounded in your data rather than in its training. You need it when answers must reflect information the model has never seen. You do not need it for tasks that are classification, extraction or transformation of text the user already supplied.
That has to be designed for rather than hoped about. Confidence thresholds, a human review path for anything consequential, and a durable record of what was generated and what was done with it. A feature with no wrong-answer path will find its own.

Sources

Page reviewed