Prompt Caching Explained: How to Reduce AI Agent Costs

You build an agent with a 10,000-token system prompt and a dozen tools. A user has a 30-turn session. At the end you look at the bill and find you paid for almost a million input tokens, even though the user typed a few hundred words.

Nothing is broken. Every turn re-sends the whole conversation, and you pay for all of it each time. Prompt caching is the provider feature that stops you paying full price for the part that did not change.

What is prompt caching?

Prompt caching lets a model provider reuse the work it already did on the beginning of your prompt, and charge you a fraction of the normal input price for those tokens. A cache hit is a prefix match on the prompt. It is not a cache of responses: the model still generates a fresh answer every time, and Anthropic states that caching has no effect on output generation.

The match has to be exact. If the first 20,000 tokens of this request are identical to the first 20,000 tokens of a recent request, those tokens can be read from the cache. Change one character near the start and everything after it is processed, and billed, as new input.

Prompt caching by provider at a glance

Prices and thresholds change often. Check the Anthropic, OpenAI, Gemini, and xAI pricing pages before you budget. xAI also lists a separate cached input price for Grok models. As one example of the Gemini numbers, the pricing page lists Gemini 3.8 Flash at $0.75 per million input tokens and $0.075 per million cached tokens through the end of 2026, plus $0.50 per million tokens per hour of explicit cache storage.

Why is my AI agent so expensive to run?

Agents are expensive because the API is stateless: every turn re-sends the system prompt, every tool definition, and the entire history so far. The cost of a session grows with the square of its length, not in a straight line.

That same shape is why agents benefit most from caching. The prompt is a long stable prefix with a small new piece on the end. The system prompt and tools never change. The history only grows. On turn 20, everything from turns 1 to 19 is byte-for-byte what you sent last time.

How much does prompt caching save?

In a 30-turn agent session, prompt caching can cut input cost by about 84 percent. Here is the arithmetic, using Anthropic's published multipliers (writes at 1.25x, reads at 0.1x) and an illustrative model price of $3 per million input tokens.

Assume a 10,000-token system prompt plus tools. Each turn adds 1,500 tokens of user message, tool results, and assistant output to the history. The session runs 30 turns, each within five minutes of the last. With caching, each turn reads the previous prompt from the cache and writes only the new 1,500 tokens.

Input cost drops from $2.99 to $0.49, about 84 percent. Notice turn 1: it costs more with caching, because you pay the write premium before any read. Caching pays for itself on the second request. Output tokens are billed the same either way, so your total bill falls by less than 84 percent.

Why is my prompt cache not hitting?

A cache miss almost always means something early in the prompt changed. Because matching is by prefix, one changed byte invalidates everything after it. The usual causes:

• A timestamp or request ID in the system prompt. "The current time is 14:03:22" makes every request unique from the first line.
• Reordered or changed tools. On Anthropic the cache is built in the order tools, system, messages, and a change to tool definitions invalidates all three levels.
• Non-deterministic serialization. If tool schemas come from a map with unstable key order, two logically identical requests produce different bytes.
• Editing earlier messages. Trimming, summarizing, or rewriting history mid-session discards the cache from the edit onward.
• Switching models. Caches are per model. Routing turn 12 to a different model starts cold.
• Waiting too long. A user who returns after the lifetime in the table above pays for a fresh write.
• Being under the minimum. Anthropic notes that prompts below the threshold are simply not cached, with no error.

How do I structure prompts for caching?

Put stable content first and volatile content last. A good order is tool definitions, then the system prompt, then long reference documents, then conversation history, then the newest message.

If the model needs the current date or user details, put them in the latest user message instead of the system prompt. Sort tool definitions and serialize them deterministically. Treat history as append-only, and when you do need to compact it, do it once at a clear boundary and accept a single miss.

On Anthropic, the simplest setup is one top-level field, which places the breakpoint on the last cacheable block and moves it forward as the conversation grows:

{
  "model": "your-model-id",
  "cache_control": { "type": "ephemeral" },
  "system": "Stable instructions...",
  "tools": [],
  "messages": []
}

On OpenAI, caching is automatic. The OpenAI guide describes an optional prompt_cache_key, which on earlier models helps route requests that share a prefix to the same cache.

How do I measure my cache hit rate?

Read the usage fields in each response and divide cached tokens by total input tokens. In the worked example above the hit rate is 942,500 out of 997,500, or 94 percent.

• Anthropic: cache_read_input_tokens, cache_creation_input_tokens, and input_tokens. Total input is the sum of all three.
• OpenAI: usage.input_tokens_details.cached_tokens, and cache_write_tokens alongside it.
• Gemini: the caching guide points to total_cached_tokens in the response usage.
• DeepSeek: prompt_cache_hit_tokens and prompt_cache_miss_tokens.
• OpenRouter: cached_tokens, cache_write_tokens, and a cache_discount value, per its caching guide.

Log these per request. A hit rate that falls off after a deploy usually means someone added something volatile to the top of the prompt.

Does prompt caching make responses faster?

Yes, for the start of the response. Cached tokens skip most of the input processing, so time to first token improves on long prompts. Generation speed for the output does not change. Anthropic also documents pre-warming the cache with a max_tokens: 0 request so the first real user message does not pay the miss.

Does prompt caching work through OpenRouter or a gateway?

Yes, but the gateway has to keep you on the same upstream provider. OpenRouter documents that it passes cache_control breakpoints through for Anthropic and Gemini models, relies on automatic caching for OpenAI, DeepSeek, and Grok, and routes follow-up requests for the same model to the provider that served the cached request. A gateway that load-balances each request to a different host will miss every time.

When does prompt caching not help?

Caching does not help when nothing repeats. Single-shot requests, prompts below the minimum size, traffic spaced further apart than the cache lifetime, and prompts whose opening changes on every call all miss. On providers that charge for writes, a prefix that is written and never read costs slightly more than no caching. Output-heavy work such as long generations sees little benefit, because caching only discounts input.

How does Dexto handle prompt caching?

Dexto runs the agent loop for you, so the prompt structure described above is the harness's job and not yours. You pick a model in the model catalog and work; the system prompt, skills, connected tools, and history are assembled by Dexto on every turn. This is a large part of what we mean by harness engineering.

Three things are specific and checkable:

• Cache markers for Claude models. When Dexto sends a request with reasoning settings to an Anthropic model, it sets the ephemeral cache_control option on the request. Providers with automatic caching need no marker.
• Cached tokens are tracked separately. Dexto reads the cached-token count from each model response and records cache read and cache write tokens next to input, output, and reasoning tokens. The usage breakdown on a message in the app shows "Cache read" and "Cache write" when they are non-zero, so you can see your hit rate without writing logging code.
• Cache pricing is part of the catalog. Dexto's model registry carries cache read and cache write prices per model where the provider publishes them. For usage billed through Dexto, the charge is taken from the cost the upstream gateway reports for that request, which already reflects cache discounts.

Because caches are per model, this also affects how you choose a model: switching models mid-session is a cold start, so switch between tasks and not between turns. The same prefix logic applies to skills and MCP tools. A stable set of connected tools keeps the front of the prompt identical from turn to turn. If you bring your own provider key, the provider's own caching rules and prices from the table above apply to your account.