Writings

Prompt order decides your inference bill

How prefix caching works, what quietly breaks it, and why prompt layout is a product decision.

10 June 2026Prefix caching

Two teams can ship the same feature, on the same model, at the same volume, and see a wide gap in what it costs to run. Often the content is identical and only the order differs.

How prefix caching works

The provider caches the computed model state for a prompt prefix so it doesn't get recomputed on the next request. The internal state, not the output.

Matching is exact and byte-level. Change one token anywhere in the cached prefix and everything from that point down invalidates.

Provider specifics differ:

  • Anthropic builds the prefix as tools, then system, then messages. You mark the end of stable content with cache_control and one entry gets written, a hash of the prefix ending exactly at that breakpoint. Nothing is written for earlier positions.
  • OpenAI orders system first, then tools. On newer models you pass prompt_cache_key consistently across requests sharing a prefix, and can set an explicit breakpoint so the changing suffix doesn't get written.
  • Gemini offers implicit caching, on by default above a minimum request size, and explicit caching where you create a cache object with a TTL and reference it.

Cache reads cost a fraction of normal input tokens. Cache writes cost more than normal input tokens. On Anthropic's pricing, a write runs 1.25x the base input rate and a read runs 0.1x, which puts break-even at two reads per write.1 A prefix written and read once loses money.

What breaks it

Teams turn caching on, watch hit rate sit under 10%, and conclude the feature is broken. Usually something volatile is sitting near the top of the prompt:

  • a timestamp in the system prompt
  • a session or request ID
  • tenant or user name interpolated into instructions
  • rotating or A/B-tested instruction variants
  • retrieved chunks injected above the static block

Any one of those invalidates everything below it on every request.

There's a quieter failure worth knowing about. Rust and Swift randomise map key order when serialising JSON, so your tool definitions are identical in your code and different on the wire every request, and nothing ever matches.

A couple of invalidators aren't visible from reading the prompt: changing tool_choice, and adding or removing an image anywhere in it.

Before touching any caching config, map which parts of your prompt actually stay constant across requests.

Order by rate of change

stacked by how often it changes. stable at the top. tool definitions never changes within a version cached system instructions, examples never changes within a version cached stable corpus, retrieved once changes per session at most cached cache breakpoint hash of everything above session context: locale, tenant, timezone changes per session conversation history appends each turn. safe. editing it is not. tool results, current snapshot, user turn changes every request recomputed move any dashed row above the line and every request below it recomputes from that point down

Stack content by volatility, most stable at the top:

  1. Tool definitions
  2. System instructions and few-shot examples
  3. Stable corpus, retrieved once per session
  4. breakpoint
  5. Session context: locale, tenant, timezone
  6. Conversation history
  7. Tool results, current snapshot, user turn

Retrieval placement is where this usually breaks. Retrieve per-turn and insert results above your instructions and every request misses.

Two ways out. Move retrieval below the stable block, which keeps the cache but means the model sees retrieved content late. Or cache a broader corpus once and let the model narrow inside it, which trades input tokens for cache hits and is often cheaper at volume.

Two things that look like good engineering

Compaction. Summarising conversation history to save tokens rewrites the prefix, and everything from the edit point down invalidates. You save tokens on that request, pay full prefill on content you'd already cached, and keep paying until the new prefix builds hits back up. Append-only history caches well, edited history doesn't. There's research on this specifically: aggressive truncation minimises per-turn token count while shattering prefix continuity, and the prefill penalty can exceed the token savings.2

Caching the full context. Cache everything including tool results and you pay write costs on session-specific content that will never be read again. On GPT-4o this reversed the latency improvement that system-prompt-only caching had produced.3

The strategy that tests better is breaking the prefix on purpose: a breakpoint after the system prompt, and after each tool result.

Measuring it

Four numbers worth watching:

  • Cache hit rate, broken out by request type. An aggregate will hide a single flow that misses every time.
  • Tokens written vs tokens read. Writes above reads means you're paying the premium without collecting.
  • Time to first token, cached vs uncached. This is where users feel it.
  • Cost per request, per flow.

Most providers expose cache diagnostics that compare consecutive requests and report which block diverged. That's the fastest way to find whatever is quietly busting your prefix.

Healthy looks like: hit rate above 70% on repeat flows, reads well above writes, a visible TTFT gap on cached calls. Broken looks like: hit rate under 10%, writes climbing every request, no TTFT difference at all.

Why this belongs in product conversations

Cache hit rate decides which features are affordable to run.

Verifying an output on every turn is either a sensible default or an unshippable cost, depending on whether the verification step reuses a cached prefix. Same with running two cheap passes and comparing them, which is one of the more reliable quality mechanisms available and gets cut in most cost reviews.

Both are quality decisions getting made on cost grounds, and most of that cost is prompt layout.