Prompt caching explained: examples, KV vs prefix caching, OpenAI vs Claude, RAG

Prompt caching lets an LLM provider reuse work it already did on a repeated prompt prefix, such as a long system prompt, tool definitions or a document, so later requests are faster and cheaper. Every major provider now offers some form of it. I build RAG apps such as RAG.NextUpgrad that stream cited answers over long context, so this page covers how prompt caching works, an example, a tutorial, KV, prefix and context caching, implicit versus explicit caching, OpenAI versus Claude, RAG and best practices.

What is prompt caching and how does it work?

Prompt caching stores the model's processed state for the start of a prompt, its key and value tensors, so a later request beginning with exactly the same tokens skips recomputing them. You pay less for those tokens and get the first token sooner.

Also asked as: prompt caching: a complete technical guide · prompt caching tutorial

The match is a prefix match: the cache helps only up to the first token that differs, so anything that changes between requests should come last. Providers set a minimum cacheable length, often around a thousand tokens, and keep entries for a limited time, typically minutes, refreshed whenever they are used. Caches are scoped to your organisation, so other customers never read your prompts. The benefit grows with prompt length and repetition, which makes caching most valuable for chat assistants, agents with large tool lists and document question answering.

What is an example of prompt caching?

A support bot sends the same long system prompt and product manual with every user question. With caching, the first request processes and stores that shared prefix, and every later question within the cache lifetime reuses it, paying full rate only for the new question.

Multi-turn chat is another example: each turn resends the whole conversation, so caching the history means each new turn processes only the latest messages. Coding agents such as Claude Code benefit heavily, since they resend large tool definitions, instructions and file contents on every step of a loop, so keeping that prefix stable across a session matters. In each case, the usage fields in the response report how many input tokens were read from cache, which confirms the cache is actually hitting.

How do you set up prompt caching step by step?

Order your prompt from most stable to most variable: tools, system instructions, reference documents, conversation history, then the new user message. Then enable caching for your provider and measure cache hits in the usage data.

On OpenAI, caching is automatic for long prompts, so the work is mostly ordering and keeping prefixes identical. On Claude, add a cache_control breakpoint to the last block you want cached, such as the end of the system prompt or a document, and add further breakpoints for content that changes at different rates. On Gemini, rely on implicit caching or create an explicit cache with a time to live. Send two identical requests a few seconds apart and check the cached token count in the second response. If hits stay at zero, remove timestamps, random IDs and reordered tools from the prefix.

What is the difference between KV caching and prompt caching?

KV caching happens inside a single request: during generation, the model stores the key and value tensors of earlier tokens so each new token attends to them without recomputing. Prompt caching keeps those tensors for a prompt prefix across separate requests.

Also asked as: kv caching vs prompt caching

Every modern inference engine uses a KV cache; without it, generating each token would reprocess the entire sequence. Its size grows with context length and batch size, which is why long contexts consume so much GPU memory. Prompt caching builds on the same data: instead of discarding the KV cache when a request ends, the server keeps the prefix portion and reuses it when another request starts with the same tokens. So KV caching speeds up generation within one answer, while prompt caching speeds up the prefill stage across many answers.

What is the difference between prefix caching and prompt caching?

They are largely the same idea under different names. Prefix caching is the term inference engines such as vLLM and SGLang use, while prompt caching is the name API providers give the billed feature built on it.

Also asked as: prefix caching vs prompt caching

In vLLM, automatic prefix caching splits the KV cache into fixed-size blocks, as in PagedAttention, and hashes each block together with the tokens before it, so any new request sharing those blocks reuses them. SGLang's RadixAttention stores prefixes in a radix tree for similar sharing across many requests. Hosted APIs such as OpenAI and Anthropic expose the same mechanism with pricing, minimum lengths and expiry rules. If you self-host open models, enable prefix caching in your server; if you call an API, follow the provider's caching rules.

What is the difference between context caching and prompt caching?

Context caching is Google's name for prompt caching in the Gemini API and Vertex AI. It comes in two forms: implicit caching, applied automatically, and explicit caching, where you create a named cache of content and reference it in later requests.

Also asked as: context caching vs prompt caching

Explicit context caches have a time to live you choose and are billed for storage while they exist, in addition to discounted reads. That suits large, reused material, such as a long video, a codebase or a set of manuals that many queries will reference over an hour or a day. Implicit caching needs no setup but gives no guarantee of a hit. Functionally, both reuse processed tokens just like other providers' prompt caching; the differences lie in control, lifetime and how storage is charged.

What is the difference between implicit and explicit prompt caching?

Implicit caching is automatic: the provider detects repeated prefixes and discounts them without code changes. Explicit caching requires you to mark what to cache, such as Claude's cache_control breakpoints or Gemini's cache objects, giving more control over hits and lifetime.

Also asked as: implicit vs explicit prompt caching

OpenAI's caching is implicit, applying to prompts above a minimum length, and Gemini offers implicit caching on recent models. Anthropic's caching is explicit, with a five-minute default lifetime and a longer one-hour option. Implicit caching is easier but less predictable, since request routing and eviction decide whether you hit. Explicit caching lets you keep a large document cached through a busy period, at the cost of managing breakpoints and, for Gemini, paying storage. Choose based on how predictable your traffic is.

How does OpenAI's prompt caching compare with Claude caching?

OpenAI caches automatically for prompts above a minimum length, discounting cached input with no code changes. Claude requires explicit cache_control breakpoints, charges a little extra to write the cache, and gives a deeper discount on cache reads.

Also asked as: openai's prompt caching vs claude caching

In practice, OpenAI is simpler: order prompts well and caching happens, though hits depend on routing, and a prompt_cache_key parameter can improve them for shared prefixes. Claude gives more control: you choose exactly where caches end, can set up to four breakpoints, and can pick a longer lifetime for infrequent traffic. Both report cached tokens in usage fields, and both require exact prefix matches. For apps that call several providers, such as RAG.NextUpgrad with its multi-provider fallback, one stable prompt layout lets caching work whichever provider answers.

How does RAG compare with prompt caching?

They solve different problems and work well together. RAG retrieves a few relevant passages per question from a large corpus, while prompt caching makes it cheaper to resend the same large context repeatedly.

Also asked as: rag vs prompt caching

If your knowledge fits in the context window and many questions hit the same material, such as one handbook, caching the whole document can replace retrieval and avoid missed chunks. For large or changing corpora, RAG is still required, since you cannot cache millions of documents in one prompt. Combining them is common: cache the system prompt, tool definitions and stable reference material, then append retrieved passages and the question at the end. Retrieved chunks vary by query, so placing them early breaks the prefix. In a pipeline with hybrid retrieval, reranking and cited streaming, the stable instructions are the cacheable part.

What are prompt caching best practices?

Put stable content first and variable content last, keep cached prefixes byte-for-byte identical, meet the minimum cacheable length, and monitor cache hit rates from usage fields. Small changes such as a timestamp at the top silently disable caching.

Keep tool definitions in a fixed order and avoid editing system prompts mid-session. Put dates, user names and retrieved passages after the cached block. For explicit caching, place breakpoints where content changes rate, such as after tools and after a document. Choose cache lifetimes that match traffic, since a short lifetime with sparse requests means paying to write caches that expire unused. Track cached tokens, time to first token and cost per request, and alert when hit rates drop after a deploy. Treat prompt layout as part of your API contract.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail