Prompt caching explained: stop paying twice for the same context

The problem caching solves

Say you've built an assistant that answers questions about your product catalogue. Every question sends the same five-page catalogue plus your instructions, then a one-line question. Without caching, you pay full price for the catalogue every single time.

With caching, the provider stores the repeated start of the request for a short time. Later requests that start with exactly the same text read it from the cache at a much lower rate, and usually get an answer faster too.

How it works with the main providers

Anthropic, OpenAI and Google all offer prompt caching on their APIs, but the details differ. Some apply it automatically once a prompt is long enough, others need you to mark which part to cache. Most charge a little extra to write to the cache and much less to read from it, and the cache only lasts for a few minutes unless it's used again. [Confirm current cache rates and durations for each provider before publishing.]

The key rule is the same everywhere: the cached part has to be identical and at the start of the request. Put fixed instructions and documents first and the changing question last.

When it's worth it

Caching pays off when a large, fixed block of text is sent many times in a short period: customer service assistants, document Q&A, coding tools and any workflow that runs through a batch of items with the same instructions.

It does little for one-off requests or prompts that change every time. In those cases, shorter prompts and a cheaper model will save more.

How to check it's working

The API response tells you how many tokens were read from the cache. Check this in your logs after setting it up. If the number is zero, something at the start of your prompt, such as a date or a user name, is changing each time and breaking the match.

How prompt caching works

When a large language model reads your prompt, it does a lot of work on every token before it starts writing an answer. Prompt caching stores the result of that work for the start of the prompt, often called the prompt prefix. Technically, the provider keeps the model's key value calculations in what's known as a KV cache, so it doesn't have to redo them.

The first request pays to write the prefix to the cache. Subsequent requests that begin with the same prefix get a cache hit and read the cached prefix instead of processing it again. If anything in the prefix is different, you get a cache miss and pay the normal rate for the entire prompt.

So the rule that matters is simple: the cached part must be identical and must come first. The changing part, such as the new user message, goes at the end.

What you can cache

Anything that stays the same across requests is a candidate: the system prompt and system instructions, tool definitions, long documents, product catalogues, policies and examples. In multi turn conversations, earlier conversation messages can be cached too, so each follow up request only pays full price for the newest part.

What breaks the cache

A cache miss usually means something near the start of the prompt text changed. Common culprits are a timestamp or today's date in the system prompt, a user's name placed before the instructions, tools added or removed between calls, or switching models mid conversation. Each model has its own cache, so changing model means starting again.

Claude prompt caching

Anthropic's Claude API lets you mark exactly what to cache. You add a cache control setting to a content block, and that point becomes a cache breakpoint. Everything up to the breakpoint is cached as one prefix. You can use explicit breakpoints at several points, for example after the tool definitions, after a long document and after the conversation history, so different parts can be reused separately.

Anthropic also offers automatic caching, which places the breakpoint for you on the last cacheable block and moves it forward as a conversation grows. That's the simplest way to enable prompt caching for chat style tools. There's a minimum cacheable prompt length that varies by model, so very short prompts won't be cached at all. [Confirm current minimum lengths and cache lifetime options in Anthropic's docs before publishing.]

Claude Code, Anthropic's coding tool, relies heavily on prompt caching. Its instructions, tool definitions and project context stay the same through a session, so most of each new request is read from the cache. This is a good example of why caching matters so much for coding assistants and other tools with long, repeated context.

OpenAI prompt caching

OpenAI applies prompt caching automatically on supported models once a prompt passes a minimum length. You don't need to mark breakpoints. The system looks for the longest prefix it has already seen recently and reuses it, which is why the same structure rule applies: static content first, variable content last.

OpenAI also lets you pass a cache key with an API request so that similar requests are more likely to land on the same cache. This can help cache hit rates when many users share the same instructions. [Confirm current OpenAI cache pricing, any write charges and how long entries last before publishing.]

Gemini context caching

Google calls it context caching and offers two kinds. Implicit caching works automatically on supported Gemini models: if your request shares a prefix with a recent one, you may get a discount on the cached part without doing anything.

Explicit caching lets you create a cache yourself, for example from a long document or video, and refer to it in later requests. You choose a cache TTL, which is how long it's kept, and you pay a storage charge for the time it exists as well as a reduced rate when it's read. That makes explicit caching useful for content you know you'll use many times, but it needs managing so you don't pay to store caches nobody uses. [Confirm current Gemini cache storage rates and TTL rules before publishing.]

What prompt caching costs

Pricing is set relative to the base input price of each model. In general, a cache write charge is a little higher than the base input price, and cache reads are much lower. Uncached input tokens, such as the new question at the end, are charged at the normal rate. Output tokens aren't affected by caching.

To work out the saving, split total input tokens into three parts: tokens written to the cache, tokens read from the cache and uncached tokens. Multiply each by its rate, which for each provider is [current cache write rate], [current cache read rate] and [current base input rate] per million tokens. Compare that with what the same requests would cost without caching.

A simple worked example

Say your support assistant sends a 10,000-token catalogue and instructions with every question, and answers 60 questions an hour. Without caching, every question pays the base input price for the whole catalogue. With caching, the first request pays the write charge, and every following request in the cache lifetime pays the much lower read rate for the catalogue plus normal price for the short question. The more requests that reuse the same cached content, the bigger the saving.

Building a caching strategy

Good caching is mostly about how you structure each prompt. These habits make the biggest difference.

• Put static content first: tool definitions, system prompt, then fixed documents
• Put anything that changes, such as dates, names and the user's question, after the cached part
• Keep the same tools available for a whole conversation rather than adding and removing them
• Stick to one model per conversation, because each model has its own cache
• Add new information as a new message rather than editing the instructions
• Group repetitive tasks so requests with the same prefix run close together

Cache lifetime and keeping a warm cache

Cache entries don't last for ever. Each provider keeps a cached prefix for a minimum lifetime, and many reset that timer each time the cache is read. If no request uses it within that time, the entry expires and the next request has to write it again. [Confirm current cache lifetimes for each provider before publishing.]

For busy tools, the cache stays warm on its own. For quieter ones, some providers offer a longer cache lifetime at a higher write price. Whether that's worth it depends on how often requests arrive. A cache that keeps expiring between requests can cost more than no cache at all, because you keep paying the write charge.

How to measure cache hit rates

Every provider reports caching in the API response. Look at the usage fields: they show how many tokens were written to the cache, how many were read from it and how many were processed normally. The names differ by provider, but the idea is the same.

Log these numbers for a few days and work out your cache hit rate: the share of input tokens read from the cache. If it's low, look at the start of your prompts for anything that changes between calls. Fixing one stray timestamp can turn a stream of cache misses into hits.

Prompt caching and your data

Caching stores a processed version of your prompt on the provider's systems for a short time. The main providers state that caches aren't shared between different customer organisations, but check each provider's data policy and your own UK GDPR position before caching documents that contain personal data. [Confirm each provider's current cache isolation and data retention wording before publishing.]