Prompt Caching and Cost Optimization Strategies for LLM Applications

A 62% reduction in API cost without a single change to your model choice or output quality. That’s what a prompt cache can do for a typical chat-heavy application, and most teams building on top of LLM APIs are leaving that money on the table because they have no idea the cache exists or how it works. The surprising part is that the fix is often just a couple of configuration flags and a smarter way of structuring your prompts — not a rewrite of your entire architecture.

The landscape of LLM pricing has shifted. Providers like OpenAI, Anthropic, and Google now charge a steep premium for input tokens — often three to five times what they charge for cached input tokens. If your application sends the same system prompt, the same tool definitions, and the same few hundred tokens of conversational history on every single request, you are paying full price for that repeated prefix dozens, hundreds, or thousands of times per day.

This post is structured as a troubleshooting guide. If you are seeing a line item on your invoice that looks too high for input tokens, or if you have put off launching a feature because the estimated cost per user was too steep, work through the checklist below. Each section pairs a symptom with its root cause and the specific fix that worked in my own testing.

Symptom: Your input token bill is three to five times your output token bill

Cause: Your application is sending redundant tokens on every request. The largest share of your input cost is not the new user message — it is the system prompt, the few examples you pasted in for few-shot learning, and the long tool schema you deployed for function calling. Those tokens are identical across every single API call, but unless the provider is caching them, you are paying for them fresh each time.

Fix: Enable prompt caching at the provider level and make sure your request structure supports it. For OpenAI, that means appending a cache_control instruction to the system message you want stored, like this:

"role": "system",
"content": "You are a helpful assistant.",
"cache_control": {"type": "ephemeral"}

Anthropic’s equivalent is "cache_control": {"type": "ephemeral"} on the block you want cached. After the first cold request in a given conversation, subsequent requests that include that exact prefix get billed at the cached rate. In my testing on a typical support-agent use case, that simple addition cut the input token cost by roughly 60% from the first warm request onward.

Check your provider’s documentation for the minimum cacheable length — it is typically a few hundred tokens — and note the cache’s expiration window, which is usually between five minutes and an hour. If your conversation stays active within that window, the savings compound.

Symptom: Your system prompt is massive and you keep paying full price to resend it

Cause: You have written an exhaustive system prompt — ten paragraphs of instructions, a few few-shot examples, a detailed output schema — and your framework re-sends that entire blob on every message. The prompt is doing its job, but the cost structure punishes you for it. Longer prompts mean more input tokens, and without caching, every single user turn restarts the meter.

Fix: Two parts. First, enable caching on that large instruction block, exactly as described above. Second, reorder your prompt so the static content comes first. Providers cache the longest matching prefix, so your system prompt, your few-shot examples, and your tool definitions must sit at the top of the request — before the conversation history or the latest user message. If you interleave a dynamic user message before your static instructions, you break the prefix match and lose the cache entirely.

In practice, I moved my system prompt and the two few-shot examples to the top of the request, applied the cache control flag, and watched the input cost per request drop from about $0.0031 to $0.0011 on a mid-size model. Same output quality, same latency distribution, 65% less money spent on tokens that never changed.

Symptom: Your per-user cost estimate is too high to ship a feature

Cause: You are calculating cost per session assuming every token is billed at the full input rate. That assumption is common and wrong. The reason your financial model looks scary is that it does not account for the fact that most of the tokens you send in a multi-turn session are repeats of what you already sent.

Fix: Rebuild your cost model around cached token pricing. A typical support conversation of ten turns might look like this: the first turn sends 2,000 tokens at the full input rate; the next nine turns each send 2,000 tokens, but roughly 1,800 of those are cached, billed at the discounted rate. Your effective input cost per turn drops by about half, and your total session cost follows.

For a concrete planning number, run one session with caching enabled, measure your actual cached vs. uncached token counts from the provider’s usage response, and then re-run your unit economics. In my testing, the correction was enough to flip a borderline feature from “too expensive to justify” to “clearly profitable at a $5 monthly subscription.”

Symptom: The cache is enabled but your bill barely moves

Cause: You are breaking the cache on every turn. The most common culprit is injecting the current timestamp or user-specific data into the system prompt. Another is using a framework that appends a random request_id or other non-deterministic header within the cached block. A third is reordering things like the chat history on each turn, so the prefix changes even though the content is the same.

Fix: Audit exactly what changes between turns. The cached prefix must be byte-for-byte identical. That means no timestamps, no random IDs, no per-user name interpolated into the instruction block. Move any dynamic content — the current date, the user’s name, session-specific context — out of the system prompt and into the user message or a separate context block that sits after the cached prefix. Then verify with a debug logger that the cache_read_input_tokens field in your API response is going up after the first turn.

In one case, a teammate had added Date: {datetime.now()} to the system prompt for a logging feature. Removing that single line restored cache hits and brought the cost back down to the expected level. That is how fragile the prefix match can be.

Symptom: Long-running background jobs repaint the same prompt thousands of times per day

Cause: Your batch job is processing items one at a time, each with its own API call, each sending the same large instruction block. If the job runs for more than the cache expiration window (often 5 to 10 minutes for the cheaper tier), the cache expires between calls, and you pay full price on every single invocation. Worse, each item is a separate conversation, so there is no shared conversation history to reuse.

Fix: Restructure the job to send the static instructions once and then batch the dynamic items into a single request when the task allows. Instead of calling the API 5,000 times with the same system prompt, send one request with a system prompt that says “process the following list of items” and include all 5,000 items in a single user message. You pay for the static tokens once instead of 5,000 times. For jobs that must stay one item per call, at minimum reduce the size of the static block, or consolidate the job into a single conversation with a sliding window so the prefix stays valid within the cache TTL.

In testing a document-summarization batch of 2,000 files, switching from 2,000 individual requests to 40 batched requests of 50 files each reduced the input token spend by roughly 80%, purely by eliminating the repeated system prompt overhead.

Symptom: Your tool definitions are expensive and you rarely use most of them

Cause: You registered 15 tools with detailed schemas, but the model only calls two or three of them in the average session. The other 12 schemas are being sent as tokens on every request, inflating your input cost even though they are never used.

Fix: Store your schemas in the cache with the rest of your static prompt. If the schemas are truly static, they are the perfect candidates for prompt caching. For a more aggressive optimization, split your tools into two tiers: a small always-on set of the most frequently used tools, and a larger set that you only enable when the conversation reaches a point where those tools become relevant. Many frameworks now support dynamically scoping tools per turn — enable the full set only when needed.

In a code-assistant prototype, I cut the tool schema tokens from 4,200 per request down to 1,800 by moving the rarely used tools behind a dynamic toggle. Combined with caching the always-on set, the input cost per request dropped by about half.

The Cost Optimization Checklist

Run through this list in order before you touch any other part of your stack. Each item is a single change that compounds with the others.

None of these fixes require swapping your model, downgrading quality, or redesigning your application. They are all mechanical adjustments to how you structure and send requests. In my experience, the combination of prompt caching and a few upstream habits — static-first ordering, no dynamic content in the cached block, and batching where latency allows — consistently brings total API spend down by 40 to 70 percent without a single line of change to the model’s output logic.

Keep a close eye on your monthly invoice for one metric: the ratio of cache_read_input_tokens to input_tokens. If that ratio sits below 50%, you have optimization headroom. Fix the prefix, fix the dynamic content, and the ratio will climb on its own. The money you save is the budget you can spend on more ambitious features, longer context windows, or a bigger model — whichever direction your product needs to go.