Where Your AI Bill Actually Goes (and How to Cut It Without Switching Vendors)

Your AI bill is bigger than it should be, and switching provider will not fix it. Most of the overspend is self-inflicted plumbing: tokens you re-send on every turn, retries you never cap, a frontier model doing work a cheap one would do, and two standing discounts you never turned on. The reason it goes unnoticed is duller than any of the drivers. Almost nobody measures token spend per feature, so the waste is invisible until the invoice arrives as one large number with no breakdown.

LeadDev’s piece on the same problem opens on a debugging session that burned through a surprising amount of budget because the model was handed a whole log file when three lines mattered. That is the shape of the whole issue. The model did exactly what it was told; it read everything it was given, and everything it was given was billed. The fix is not a better model or a cheaper vendor. It is sending less, sending it once, and knowing which feature is spending what.

This is a commentary piece, so here is the position up front: the levers that cut an AI bill are well known and mostly free to pull, but pulling the wrong one first wastes the effort. Measure before you optimize, because you cannot cut a bill you cannot itemize.

The meter runs on tokens, and you control most of them

An LLM invoice is a token meter. You are billed for input tokens (everything you send: the prompt, the system message, tool definitions, retrieved context, the running conversation) and for output tokens (everything the model generates). Output is priced higher than input on every major provider, because it is produced one token at a time rather than read in a single pass. That asymmetry matters more than it looks, and we will come back to it.

The first thing worth internalizing is that the input side is almost entirely under your control, and it is usually where the waste lives. Four drivers account for most of it.

Context dumping. The most common one, and the one from the LeadDev example. You send the entire file, the entire schema, the entire prior conversation, when the task needs a slice. Every token you paste is billed, and in a multi-turn exchange it is billed again on every subsequent turn, because the model has no memory between calls — you re-send the history each time. Handing the model a 4,000-token log to answer a question that lived in three lines is not a rounding error. It is the base rate of the whole conversation, multiplied by every turn that follows.

Verbose encoding. You pay to pretty-print. Indented JSON, redundant keys, and boilerplate framing all cost tokens without adding information the model uses. LeadDev cites roughly 30% input savings from compacting payloads without dropping any data — treat the exact figure as illustrative rather than a benchmark, but the direction is real. If you are sending structured data to a model at volume, the formatting is a line item.

No prompt caching. This is the one most teams leave on the table. If your requests share a stable prefix — a long system prompt, a fixed set of tool definitions, a retrieved document you ask several questions about — that prefix is re-billed at full input price on every single call unless you cache it. With caching on, the cached portion is dramatically cheaper: Anthropic bills a cache read at roughly 10% of the base input rate, and OpenAI applies about a 50% discount to the cached portion automatically. The catch is that caching only helps a prefix that actually recurs and stays at the front of the prompt. Put the stable material first, the variable material last, and keep the cache warm.

Reasoning tokens. Reasoning models emit internal thinking tokens the user never sees, and those are billed at the output rate — the expensive one. A three-sentence visible answer can carry thousands of billed thinking tokens underneath it. This is not a defect; it is what the model is for on hard problems. It becomes waste when you point a reasoning model at a task that never needed one, and pay output rates for deliberation on a formatting job.

The lever nobody prices in: the retry and the agent loop

Single-call cost is easy to reason about. Agent loops are where bills quietly detonate.

An agent that plans, calls a tool, reads the result, and plans again does not start fresh on each step. It re-sends the accumulated conversation — every prior step, every tool output — as input on the next call. Token cost per step therefore grows with the length of the history, and total cost for a run grows faster than linearly in the number of steps. A twenty-step loop does not cost twenty times a single step. It costs more, because step twenty is paying to re-read everything steps one through nineteen produced.

Retries make this worse in a way that is easy to miss. When a step fails and the agent retries, the naive implementation appends the failure to the history and tries again — so the retry is not cheaper than the original call, it is more expensive, because it now carries the failed attempt too. Failures also tend to cluster: a confused context produces more confusion, so a loop that has gone wrong keeps going wrong while the token meter runs at its highest per-step rate. The mitigation is unglamorous. Cap the number of steps. Trim or summarize history between steps instead of accumulating it. On a retry, reset the context rather than appending to a poisoned one. And put a hard token budget on the loop so a runaway run trips a breaker instead of a billing alert.

The specific dollar figures that circulate for these incidents — a particular monthly bill, a “200x” spike, a headline about an enterprise burning through an eight-figure sum in a month — mostly trace back to secondary blogs and are not worth repeating as fact. The mechanism is the point, and the mechanism is real: unbounded loops and blind retries turn a linear-looking cost into a superlinear one.

Two discounts you are probably not using

Before any clever optimization, there are two standing discounts that require almost no engineering and that a lot of teams simply never switch on.

Batch processing. Both major providers offer roughly 50% off for work you submit asynchronously and collect within a completion window, up to 24 hours. Anything that is not waiting on a human — evals, backfills, nightly reports, bulk classification, dataset labeling — is a candidate. If half your token volume is interactive and half is scheduled, and the scheduled half runs on the interactive endpoint, you are paying double for the part of the workload that could not care less about latency.

Right-sizing the model. The spread between a small model and a frontier model is one to two orders of magnitude per token. Running every request through the top model means paying frontier rates for classification, extraction, routing, and formatting — tasks a cheap model handles at near-parity. The research on routing is encouraging: RouteLLM, a peer-reviewed system, retained around 95% of GPT-4-level quality on its benchmarks while sending most traffic to a far cheaper model. Your mileage depends entirely on your traffic mix, which is exactly why this is not a set-and-forget switch — routing needs per-task benchmarking to know where the cheap model holds and where it drops. But the ceiling is high, and the default of “everything through the best model” is almost never the cost-optimal one.

The tooling layer: what actually cuts the token count

The levers above are behaviours. This section is the components that implement them, because “send less” is advice and “put a retriever in front of the model” is a change you can schedule. Roughly in the order they pay off.

Retrieval instead of the whole document. The single largest input-side saving available to most teams, and the direct fix for context dumping. Rather than pasting a file, a schema, or a policy document into the prompt, index it once and send only the passages the question needs. A 4,000-token document becomes three retrieved chunks. The saving compounds in a conversation, because you were re-sending that document on every turn. Ordinary vector RAG is enough for “find the relevant part”; you do not need anything more sophisticated to capture most of this.

Graph retrieval when the question is about relationships. Vector search answers “what text is similar to my question.” It is weak on “what connects to what,” so teams compensate by widening the retrieval window until the prompt is large again. Graph-based retrieval indexes entities and their relationships and pulls a relevant subgraph instead. LightRAG is the reference implementation and was designed explicitly around the cost problem: Microsoft’s GraphRAG builds precomputed community summaries during indexing, which is expensive, while LightRAG uses dual-level retrieval directly over the graph and updates incrementally rather than re-indexing. The efficiency claims circulating for it are dramatic — orders-of-magnitude reductions in indexing tokens and per-query cost — but they come from the paper and from vendor comparisons rather than independent replication. Take the architecture seriously and measure the numbers yourself.

A precomputed map instead of exploratory reading. This one is specific to agents working over a codebase or a large corpus, and it is badly underused. An agent asked “where does the sitemap get configured” will grep, open several files, read them in full, and burn tens of thousands of input tokens establishing what a static index could have told it in a hundred. Building that index once — a knowledge graph of files, symbols, and dependencies that the agent queries first — converts repeated exploration into a lookup. The cost moves from every session to one indexing run, and it is incremental after that. The same logic covers precomputed summaries, symbol maps, and API digests.

Prompt compression. LLMLingua, from Microsoft Research, drops tokens a model does not need to reconstruct the meaning, and reports compression in the 2–5x range for typical prompts with much higher ratios on redundant material. It applies where the prompt is long and unavoidably so — retrieved context, transcripts, logs. It is a real technique with a real trade-off: you are betting the discarded tokens were not load-bearing, so it wants evaluation on your own task before it goes near production.

Semantic caching. Prefix caching, covered above, only helps when requests share a literal prefix. Semantic caching goes further: two questions that mean the same thing return the same cached answer, so the second one never reaches the model. It fits support, documentation search, and internal Q&A, where the same handful of questions arrive endlessly in different words. It fits badly where answers must be fresh or personalised, and a stale hit is a correctness bug rather than a saving.

A gateway to make the rest possible. Most of this is easier behind one proxy than scattered through the application. A gateway like LiteLLM gives you a single place to tag every call, enforce per-team budgets, route by task, hold the cache, and fail over between providers. This is also the practical answer to the visibility problem from the top of the piece: per-feature attribution is a configuration change at the gateway rather than an instrumentation project across every call site.

The pattern across all six is the same. You are moving work out of the token stream — into an index, a cache, a compressor, or a cheaper model — so the expensive thing reads less.

The honest caveat: knowing the levers is not the hard part

Here is where a lot of cost-optimization advice quietly cheats. It lists the levers as if listing them were the work. It is not. Every team already knows caching and routing exist. The reason bills stay high is that pulling each lever costs engineering time, and each one only pays off for a specific usage shape.

Caching only helps a prefix that recurs; if your prompts are all novel, there is nothing to cache. Routing only pays if you benchmark each task class and accept the operational weight of running more than one model. Trimming agent history risks dropping context the next step needed. And for plenty of teams, inference is still small next to compute and storage, so it never reaches the top of anyone’s list — until it spikes, and then it reaches the top of everyone’s.

Which is why the honest first move is not caching. It is measurement. Attribute token spend to a feature, a team, or a user, the way you would attribute any other metered resource. LeadDev’s analogy is apt: cloud storage deduplicates your files internally and still bills you for the duplicates, because the vendor has no reason to itemize your waste for you. Neither does an LLM provider. The monthly invoice is one number by design. Until you break it down yourself — per feature, per endpoint, per user — you are optimizing in the dark, and the odds that your effort lands on the driver that actually matters are no better than a guess.

What to do Monday morning

Do these in order, because the order is the point.

1. Instrument spend per feature. Tag every model call with the feature or endpoint that made it. Get a breakdown, not a total. This is the step that tells you whether the rest of the list even applies to you.
2. Find your biggest single driver in that breakdown. It is usually one feature, one loop, or one oversized default. Fix that before touching anything else.
3. Turn on the free discounts. Enable prompt caching on your stable prefixes; move non-interactive workloads to the batch endpoint. Neither needs a rewrite.
4. Cap the loops. Put a step limit and a token budget on every agent. Reset context on retry instead of appending.
5. Route by task, once you have the data. Send the cheap model everything it can handle; escalate to the frontier model on a signal, not by default.

None of this requires a new vendor, and none of it requires a heroic rewrite. It requires knowing where the money goes — which is the one thing the invoice will never tell you, and the one thing you can find out in an afternoon of instrumentation.

Every rung on that ladder costs engineering time that the roadmap is already spending, which is the real reason bills stay high at teams who can name each lever. A focused engagement buys that time back against one problem.

References

• LeadDev — Why your AI bill is bigger than it should be
• RouteLLM — Learning to route LLMs with preference data
• Anthropic — Prompt caching
• OpenAI — Batch API