What is prompt caching?
Prompt caching is an API feature that stores the processed form of a prompt's opening section so later requests beginning with the same tokens can skip that work. Cached input is billed at a fraction of the normal rate and comes back faster. It helps only when the prefix is identical, so stable content goes first and changing content last.
Part of Glossary: the terms behind MCP, coding agents and shared pages, defined plainly.
Order the prompt so the stable part comes first: tool definitions, system prompt, long reference material, then the conversation.
Mark where the stable part ends (Anthropic's cache_control) or let the API find the longest matching prefix itself (OpenAI, and Anthropic's automatic mode).
The first request pays a little extra to write the cache entry. Every following request whose prefix matches reads it at a fraction of the price.
A match is all or nothing. Change one character early in the prompt, swap a tool, add an image, and everything after that point is a miss.
Entries expire after minutes of inactivity unless you buy a longer lifetime, so steady traffic benefits most and an idle session starts cold.
Source: Anthropic docs: Prompt caching
Prompt caching versus the context window
Caching changes what you pay and how long you wait. It does not change what the model sees. The cached tokens are still in the request, still counted against the context window, and still part of what the model attends to. So caching is not a way to fit more in. It is a way to stop paying full price for the part of the prompt that is the same as last time.
Both large API vendors work on the same principle. The prompt has an order: tool definitions, then the system prompt, then the messages. The API hashes a prefix of that sequence and keeps the processed state for a while. A later request whose prefix hashes the same resumes from that state. Anthropic lets you mark the end of the stable part with cache_control, or set one top-level flag and let the API move the breakpoint forward as the conversation grows. OpenAI caches automatically for prompts over a minimum length and lets you add a breakpoint and a cache key. In both cases the matching is exact: the whole prefix, every character, in the same order.
Example: a coding agent's turn
A coding agent sends one request per step, and each request is the previous one plus a little more: a tool result, a new message. That is the ideal shape for caching. Turn one writes the cache for the system prompt, the tool list and the instruction files. Turn two matches all of that and pays the cheap rate for it, adding only the new tokens at the end. Over a long session, nearly every token the agent sends is a cache read.
Now break it. The agent connects a new MCP server mid-session. The tool list changes, and it sits before everything else in the prefix, so the system prompt and the whole history miss too. One change near the top invalidates everything after it. The same happens if a timestamp sits in the system prompt: a new hash on every request, and no read ever lands.
Why it matters
On current models from both vendors a cache read costs about a tenth of the normal input price, and a write costs a quarter more than normal for the standard lifetime; longer lifetimes cost more to write. For an agent that re-sends a long history on every turn, that difference is most of the bill. It also cuts latency, since the prefix is not reprocessed. The lifetime is short by default, from a few minutes to half an hour depending on the vendor, and it is refreshed each time the entry is read. An agent working steadily keeps its cache warm; one left idle over lunch starts cold.
Common mistakes
- Dynamic content first. A date, a user id or a random example near the top of the system prompt defeats the whole cache. Put anything that changes at the end.
- Rewriting history. Editing or summarising earlier turns changes the prefix. Append new messages, compact only when you must, and expect a cold cache afterwards.
- Firing parallel requests cold. The entry exists only after the first response starts. Send one request, wait for it, then send the rest.
- Reading zero cached tokens as a bug. Prompts under the minimum length are processed uncached with no error. Check the usage fields in the response to see whether a read happened.
FAQ
- How much does prompt caching save?
- On current models from both Anthropic and OpenAI, a cache read is billed at about a tenth of the normal input price, less on some models, while writing an entry costs about a quarter more than normal. For an agent that re-sends a long history every turn, that is most of the bill.
- Does prompt caching change what the model sees?
- No. The cached tokens are still in the request, still count toward the context window and still take the model's attention. Caching changes the price and the latency of the prefix, not the answer.
- Why is my prompt caching hit rate low?
- Usually something near the top changes every request: a timestamp, a user id, a tool list that varies. Other causes are parallel requests sent before the first one responds, prompts under the minimum cacheable length, and gaps longer than the cache lifetime.
Stop sending HTMLover Slack.
Create an account in a minute, connect your coding agent, and make your first board.