{"id":22805,"date":"2026-07-28T21:52:03","date_gmt":"2026-07-28T21:52:03","guid":{"rendered":"https:\/\/scannn.com\/prompt-caching-in-agents-earendil\/"},"modified":"2026-07-28T21:52:03","modified_gmt":"2026-07-28T21:52:03","slug":"prompt-caching-in-agents-earendil","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/prompt-caching-in-agents-earendil\/","title":{"rendered":"Prompt Caching In Agents | EARENDIL"},"content":{"rendered":"\n
\n

Large language models are often thought of like functions: send in some text,
\nreceive some text. That is a useful abstraction, but it ignores one of the most
\nimportant parts of running a coding agent: most of the input is the same as last
\ntime. In other words we mostly append to it.<\/p>\n

A coding agent sends the model its system prompt, tool definitions, project
\ninstructions, conversation history, tool calls, and tool results. On the next
\nturn it sends almost all of that again, plus a small amount of new material.
\nOnce a session has grown to tens or hundreds of thousands of tokens, recomputing
\nthe whole prompt for every turn is slow and expensive.<\/p>\n

Prompt caching is what makes this somewhat economic, but it is also quite
\nfragile. A changed tool definition, a model switch or a provider routing
\ndecision can turn what one would expect to be a cheap incremental request into a
\nfull replay of the context.<\/p>\n

For coding agents, cache behavior is therefore not just an implementation detail or
\noptimization. It affects latency, cost, tool design, session design, and even
\nwhich product features should be made available.<\/p>\n

What a KV Cache Contains<\/h2>\n

A transformer processes a prompt in two broad phases. During prefill<\/strong>, it
\nreads the input tokens and computes attention state for them. During decode<\/strong>,
\nit produces new tokens one at a time.<\/p>\n

At each attention layer, every processed token produces a key and a value. These
\nare not quite like key-value lookups in a hash table: both are arrays of
\nnumbers, usually floats or lower-precision quantized values. When processing a
\nnew token, the model compares that token’s query<\/strong> with the earlier keys<\/strong>
\nto determine how relevant each earlier token is. It then uses those relevance
\nscores to form a weighted mixture of the corresponding values<\/strong>. In that
\nsense, a key is what the model matches against, while a value is the information
\nit retrieves (but the lookup is fuzzy rather than “returning a single exact
\nmatch” like a dictionary lookup.)<\/p>\n

Those keys and values are retained so that the next generated token can attend
\nto everything that came before without recomputing the earlier tokens. This
\nretained state is the KV cache<\/strong>.<\/p>\n

Conceptually, a request looks like this:<\/p>\n

request 1:\n\n<\/span>[system]<\/span>[tools]<\/span>[user]<\/span>[assistant]<\/span>[tool result]<\/span>[user]\n<\/span><--------------------- prefill -------------------->\n<\/span>                       |\n<\/span>                       K and V tensors per token and layer\n\n<\/span>request 2:\n\n<\/span>[system]<\/span>[tools]<\/span>[user]<\/span>[assistant]<\/span>[tool result]<\/span>[user]<\/span>[new]\n<\/span><---------------- reusable prefix ----------------><\/span><--->\n<\/span>                                                    |\n<\/span>                                                    new work\n<\/span><\/code><\/pre>\n
The real representations are more complicated, model-specific, and “quite”
\nlarge.  The important property is that they correspond to a particular token
\nprefix.  Two prompts that mean the same thing but tokenize differently do not
\nshare a KV cache.  If a token changes in the middle, everything after that token
\nis a different continuation.<\/p>\n
Prompt caching extends the lifetime of this state beyond one generation.  When
\nthe next API request from the coding agent begins with the same tokens, the
\ninference system can reuse the stored work for the matching prefix and prefill
\nonly the new suffix.  So far, the theory.<\/p>\n
Where the Cache Lives<\/h2>\n
In order for a cache to work it needs to be stored somewhere, and it needs to be
\naddressable.  There are two broad ways inference systems make KV caches
\navailable to a later request.<\/p>\n
The simpler approach is session affinity<\/strong>.  It works by keeping the KV cache
\non or near the GPU that computed it, and routing the next request back to the same
\nworker.  A session ID or prompt-cache key becomes a trivial routing hint and so
\nyou can potentially even deal with this problem on the HTTP load balancer level
\nwithout having to look into the payload.<\/p>\n
request(session-42) <\/span>--> router <\/span>--> worker 7 <\/span>--> GPU 7 KV cache\n<\/span>next(session-42)    <\/span>--> router <\/span>--> worker 7 <\/span>--> GPU 7 KV cache\n<\/span><\/code><\/pre>\n
This avoids moving a very large cache over the network.  It is fast when it
\nworks, but it constrains scheduling.  The selected worker can become overloaded,
\nrestart, or evict the entry.  A router may also decide that balancing the fleet
\nis more important than preserving one session’s cache.  It is however a very
\nattractive solution because it works with little extra deployed infrastructure
\nand hardware.<\/p>\n
The other approach is to distribute the cache<\/strong>.  KV blocks can be stored in
\nanother memory tier or made available across workers, so a request is not tied
\nas tightly to one GPU.<\/p>\n
                         +--------------------+\n<\/span>request <\/span>--> scheduler <\/span>-->| worker 3 \/ GPU 3   |\n<\/span>             |           +--------------------+\n<\/span>             |\n<\/span>             +----------> distributed KV blocks\n<\/span>             |\n<\/span>             +----------> worker 9 \/ GPU 9\n<\/span><\/code><\/pre>\n
That improves scheduling flexibility and recovery, but moving, indexing, and
\nretaining KV blocks is itself a systems problem.  Implementations mix GPU memory,
\nhost memory, local storage, remote storage, prefix-aware routing, and eviction
\npolicies in different ways.<\/p>\n
To put KV caches into perspective: they can be large but they are in some ways
\nsmaller than one would assume.  With various tricks, the size of KV caches can
\nbe reduced to a handful of gigabytes, even for long conversations.<\/p>\n
Caches and Prefixes<\/h2>\n
Pi sessions are trees, not lists.  \/tree<\/code> can move the active conversation back
\nto an earlier point and continue along another branch.  A rewind can discard the
\nactive suffix without deleting it from the session file.  A new branch can share
\nmost of the old context, a little of it, or effectively none of it.  This design
\nis not unique to Pi, quite a few coding agents have something at least
\nconceptually similar.  Even if you do not represent the session as a tree,
\nit’s not uncommon for agents to have some form of rewinding.<\/p>\n
                             +-- E -- F  another branch\n<\/span>                             |\n<\/span>session S: root <\/span>-- A <\/span>-- B <\/span>-- C <\/span>-- D  current branch\n<\/span>                   |\n<\/span>                   +-- Z  branch near the start\n<\/span><\/code><\/pre>\n
All three branches can have the same Pi session ID.  From the router’s
\nperspective they are one session.  From the prompt cache’s perspective they are
\nthree token sequences with only partial prefix overlap.<\/p>\n
If the cache keeps reusable prefix blocks, jumping from D<\/code> to F<\/code> may still
\nreuse root -> C<\/code>.  If it only retains the hottest continuation, if the shared
\nblocks were evicted, or if the request is routed elsewhere, the hit can be much
\nsmaller.  Jumping to Z<\/code> may preserve only the system prompt and initial tool
\ndefinitions even though it starts from A<\/code>.  The precise cache management
\nbehavior here depends greatly on the providers.<\/p>\n
The reverse can also happen. \/fork<\/code> or a new session can produce a new session
\nID while carrying over a large amount of identical context.  A routing system
\nthat isolates caches by session key may fail to notice that useful overlap.<\/p>\n
The reusable prefix determines what work can be cached.  Session identity merely
\nhelps the infrastructure find likely content.  On some systems the routing key
\nis crucial to manage caches, on others it’s merely an optimization.<\/p>\n
Explicit vs Automatic Prefix Caching<\/h2>\n
Provider APIs expose caching in two main styles.<\/p>\n
Anthropic’s traditional interface uses explicit cache_control<\/code> points.  The
\nclient marks boundaries after stable parts of the request, such as the system
\nprompt, tool definitions, or the latest cacheable conversation content.  The
\nserver can then write or look up the prefix ending at those points.  The boundary
\nis explicit, but reuse still requires the content before it to match.  Not only
\nare the cache points explicit, so is the pricing.  You pay for cache writes, and
\nyou get to choose for how long which comes at different price points.<\/p>\n
Other APIs use automatic prefix caching.  The client sends the request normally,
\nand the provider finds a reusable prefix without client-placed breakpoints.  A
\nprompt-cache key or session header may improve routing or grouping, but it does
\nnot make different prefixes equal.<\/p>\n
Why Tool Loadouts Trash Caches<\/h2>\n
Tool definitions usually appear before the conversation and they are “folded”
\ninto the system prompt internally.  Their names, descriptions, and JSON schemas
\nare model input just like any other text.  Adding one tool, removing one,
\nchanging its schema, or even serializing the tools in a different order can move
\nthe first mismatch close to the start of the prompt.<\/p>\n
turn 1: <\/span>[system]<\/span>[read]<\/span>[write]<\/span>[bash]<\/span>[conversation...........]\n<\/span>turn 2: <\/span>[system]<\/span>[read]<\/span>[write]<\/span>[bash]<\/span>[deploy]<\/span>[conversation...]\n<\/span>                                   |\n<\/span>                                   old conversation is now\n<\/span>                                   after a mismatch\n<\/span><\/code><\/pre>\n
This is a common surprise with plugin systems and MCP-style tool catalogs.
\nLoading a tool only when it becomes relevant sounds efficient because fewer
\nschemas are sent initially.  On most models, however, the newly expanded loadout
\ninvalidates the cached conversation that follows it.  Saving a few tool-schema
\ntokens can cause tens of thousands of conversation tokens to be processed again.<\/p>\n
Some newer model APIs support additive tool loading<\/strong>.  A tool can become
\navailable at a specific tool result inside the transcript instead of being
\ninserted into the original tool list.  The old prefix remains unchanged:<\/p>\n
[system]<\/span>[initial tools]<\/span>[conversation]<\/span>[new tool]<\/span>[next turn]\n<\/span><--------- cached prefix ----------->\n<\/span><\/code><\/pre>\n
Pi nowadays supports this for models with native deferred-tool mechanisms.  When
\nan extension makes a purely additive change with setActiveTools()<\/code>, Pi records
\nthe added names on the tool result.  For supported Anthropic models it uses
\ndeferred definitions and a tool_reference<\/code> and for supported OpenAI models it
\nemits the corresponding tool-search items.  Other models get a safe fallback: Pi
\nsends the complete active tool list on the next request, which works
\nfunctionally but may wipe the prompt cache.<\/p>\n
The word additive<\/strong> matters as removing tools, replacing one loadout with
\nanother, or changing prompt snippets still changes earlier input.  An extension
\nthat rebuilds the system prompt, shuffles tool order, injects timestamps, or
\nchanges active tools every turn can accidentally defeat caching for the entire
\nsession.<\/p>\n
Extensibility means Pi cannot guarantee cache stability on behalf of every
\nextension.  We can provide cache-friendly mechanisms; extensions still have to
\nuse them and from what we have seen, for many extensions cache efficiency is an
\nafterthought.  This is, in part, because when you pay on a fixed subscription the
\nassociated cost with cache misses is not quite as obvious.<\/p>\n
Interruptions and TTLs<\/h2>\n
Some important prompt caches have short default lifetimes.  Anthropic’s default
\nfive-minute cache is particularly important because it is shorter than many
\nnormal coding activities.  If you go to sip a coffee when using Fable, and you
\ncome back 10 minutes later, a single “say hi” message will cost you more money
\nthan you expect.<\/p>\n
That’s because while the user may think of a coding session as continuously
\nactive, the inference provider sees a sequence of isolated requests:<\/p>\n
model request <\/span>--> run tests for 7 minutes <\/span>--> model request\n<\/span>                  no cache traffic here\n<\/span><\/code><\/pre>\n
A long build, a test suite, lunch, a meeting, or simply stopping to review a
\ndiff can outlive the cache.  The next request contains the same prompt, but the
\nstored KV state is gone and the prefix is billed again as input.<\/p>\n
Since Pi is currently not a permitted harness on Anthropic’s subscription we’re
\nfollowing the 5 minute default that Anthropic recommends for API users.  However
\nfrom looking at Claude Code’s codebase we know that for their own subscription
\nusers, they are increasing that cache timeout to one hour.  The increased cost
\nof this however often is not worth it, when you need to pay API token prices.<\/p>\n
But you can opt into this.  Some providers such as Anthropic expose longer
\nretention controls.  For supported direct APIs, Pi users can set
\nPI_CACHE_RETENTION=long<\/code> to request them.  That is still only a request: Pi
\ncannot force a gateway to retain an entry, prevent eviction under memory
\npressure, or keep a cache alive while no model request is being made.<\/p>\n
The Price of a Miss<\/h2>\n
Providers usually price uncached input, cache writes, and cache reads
\ndifferently.  Cache reads are commonly discounted because the expensive prefill
\nwork has already happened.  Cache writes can carry a premium because the provider
\nis promising to retain state for later use.<\/p>\n
Imagine a coding session with 100,000 tokens of history followed by a short new
\nrequest as the Fable example above.  When the cache works, almost all of that
\nhistory is charged at the lower cache-read price.  Only the small amount of new
\nmaterial needs to be processed at the regular input price and potentially
\nwritten to the cache.<\/p>\n
When the cache misses, the provider has to process the entire 100,000-token
\nhistory again at the regular input price.  It may also charge to write that
\nhistory back into the cache.  This is why a short request such as continue<\/code> can
\nbe surprisingly expensive after a cache expires.  In a long coding session,
\nre-reading the old input can cost much more than generating the next answer.<\/p>\n
Caching also has a chance to create non-obvious incentives.<\/p>\n
The user should want high hit rates because they reduce latency and price.  An
\ninference operator that owns the GPUs should want them too: less prefill work
\nmeans more requests served with the same hardware.  A well-designed cached-token
\ndiscount can align both sides while leaving the operator with better margins.<\/p>\n
A gateway or reseller can have a different incentive.  If it earns revenue from
\ninput tokens billed at the uncached rate, a cache miss can produce a larger
\ncustomer invoice than a hit.  Whether that also produces more profit depends on
\nits upstream costs, contracts, and who operates the cache.  In a badly aligned
\nstack, the party responsible for routing may not bear the full cost of misses,
\nwhile the party billing the user earns more revenue when they happen.<\/p>\n
That does not mean providers sabotage caches but it means cache performance
\nshould be observable.  Users should not have to infer that only from a
\nsurprisingly large bill.  Understanding if something odd is going on with caches
\ncan be an important insight.<\/p>\n
Strict cache adherence also means less flexibility for a gateway to route you to
\nthe best option in-between turns.  You might want to take a cache hit to continue
\nwith a different model which from that point onwards might be more economical,
\nor it might be the case that you might be better off load balancing to another
\nprovider.<\/p>\n
Why Pi Does Not Prune Aggressively<\/h2>\n
Now that you’ve made it this far, you probably have an idea why Pi does not prune
\ntool calls.  It is tempting to control cost by continuously deleting old tool
\nresults or rewriting history and sometimes that is necessary, especially near
\nthe context-window limit.  But as we have learned, pruning has a cache cost of
\nits own.<\/p>\n
Deleting content from the middle changes the prefix at the deletion point.  All
\nsurviving conversation after it may need to be processed again.  The immediate
\ncost of rewriting a long cached context can exceed the future savings from
\nremoving a small number of cheaply cached tokens.<\/p>\n
A rough break-even comparison is:<\/p>\n
one-time rewrite cost\n<\/span>    ~= surviving tokens after the edit * (uncached price - cache-read price)\n\n<\/span>future savings per turn\n<\/span>    ~= pruned tokens * cache-read price\n<\/span><\/code><\/pre>\n
This is not only an accounting question as old tool results often contain the
\nevidence the model used to make later decisions.  Removing them can degrade
\nbehavior even if a summary preserves the gist.<\/p>\n
Pi therefore prefers a stable, append-oriented transcript and does not treat
\nevery old token as waste.  Compaction is available when context pressure
\njustifies a lossy rewrite.  Because compaction deliberately creates new context
\nrather than accidentally re-billing an unchanged prompt, Pi treats it as a cache
\nreset rather than a cache failure in its session statistics.<\/p>\n
The goal is not the smallest possible prompt but the best trade-off among model
\ncontext, cache reuse, latency, and price.<\/p>\n
Simultaneously there can be a case for pruning too.  If you are working with
\nproviders that do not discount you for good cache usage, or it’s for whatever
\nreason not possible to get high cache rates, it might be preferable to prune.
\nIt definitely improves the opportunity for the router to balance between
\ndifferent backends as caches are not transferable.<\/p>\n
What Pi Can and Cannot Do<\/h2>\n
Pi works to keep stable inputs stable.  It passes a consistent session ID and
\nprovider-specific cache hints, places explicit cache points for APIs that
\nrequire them, records cache-read and cache-write usage, and supports
\nmessage-anchored additive tool loading where models allow it.  Its default
\ntranscript behavior also avoids gratuitously rewriting old context.<\/p>\n
Pi cannot control every layer after the request leaves the machine.  It cannot
\nchoose a provider’s eviction policy, extend a cache beyond what the API permits,
\nkeep a particular GPU alive, or guarantee that a gateway honors affinity.  It
\nalso cannot preserve a prefix that an extension changes.<\/p>\n
What it can do is make cache health visible.<\/p>\n
The interactive footer shows cumulative cache reads and writes as R<\/code> and W<\/code>,
\nplus CH<\/code> for the latest request’s cache-hit rate.  The \/session<\/code> command gives
\na fuller view: total cached and uncached input, cumulative hit rate, cost, and
\nan estimate of tokens and dollars re-billed by significant cache misses<\/a>.<\/p>\n
Messages\n<\/span>Total: 178\n<\/span>User: 6\n<\/span>Assistant: 58\n<\/span>Tools: 114 calls, 114 results\n\n<\/span>Tokens\n<\/span>Input: 7,129,883\n<\/span>  Cached: 6,776,832 (95.0%)\n<\/span>  Uncached: 353,051\n<\/span>Output: 30,013\n<\/span>Total: 7,159,896\n\n<\/span>Cost\n<\/span>Total: $6.054\n<\/span>Cache Re-billed: $0.728 (161,744 tokens, 2 misses)\n<\/span><\/code><\/pre>\n
Users who want misses called out as they happen can enable Show cache miss
\nnotices<\/strong> in \/settings<\/code>, corresponding to showCacheMissNotices<\/code> in
\nsettings.json<\/code>.  Pi then inserts a warning after a significant miss, including
\nthe estimated re-billed tokens and cost.  When it can observe a model switch or
\nan idle gap beyond the usual short TTL, it says so.  For other misses it reports
\nthe fact without pretending to know what happened inside the provider.<\/p>\n
Common Reasons for Worse Cache Performance<\/h2>\n
When a session’s cache-hit rate looks wrong, the usual causes are:<\/p>\n
    \n
  1. Idling.<\/strong> A command, review, or conversation pause exceeds the provider’s retention window.<\/li>\n
  2. Model or provider switches.<\/strong> KV state is model-specific and generally does not move across providers.<\/li>\n
  3. Branch navigation.<\/strong> \/tree<\/code>, rewinds, forks, and alternate branches can change the active token sequence even when the session ID remains the same.<\/li>\n
  4. Compaction or manual history rewriting.<\/strong> These intentionally replace part of the prompt and establish a new prefix.<\/li>\n
  5. Tool and reasoning level changes.<\/strong> Adding, removing, reordering, or editing tool definitions changes an early part of the request unless the model supports message-anchored loading and the change is purely additive. Reasoning level changes usually have the same effect.<\/li>\n
  6. Dynamic system prompts.<\/strong> Timestamps, random values, changing project context, and extension-provided prompt snippets can invalidate everything after them.<\/li>\n
  7. Extension context transforms.<\/strong> An extension that modifies old messages or provider payloads can make an apparently stable Pi transcript unstable on the wire.<\/li>\n
  8. Provider routing and eviction.<\/strong> The prompt can be identical and still miss because the relevant KV blocks are no longer available where the request lands.<\/li>\n<\/ol><\/div>\n
    Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    Large language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it. A coding […]<\/p>\n","protected":false},"author":16,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-22805","post","type-post","status-publish","format-standard","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22805","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=22805"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22805\/revisions"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=22805"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=22805"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=22805"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}