Prompt caching explained: speed, Claude, Bedrock, KV cache, prefix caching, Claude Code

Prompt caching stores the processed form of a repeated prompt prefix, such as a long system prompt, tool definitions or documents, so later requests reuse it instead of recomputing. It cuts latency and input costs substantially for repeated context. I build RAG applications such as RAG.NextUpgrad, where stable instructions precede changing questions, so this page covers speed, Claude and Bedrock setup, how caching works, KV caching, production use, Claude Code, prompt structure, paged attention and best practices.

Does prompt caching speed up inference?

Yes, for requests sharing a long identical prefix. Cached prefixes skip the expensive prefill computation, reducing time to first token noticeably, often dramatically for very long prompts, and lowering the cost of cached input tokens.

The speedup applies to processing input, not generating output, so responses with long answers still take time to write. Gains are largest when the cached part is long, such as a large document or detailed system prompt, and the changing part is short, such as a user question. The first request writes the cache and may cost slightly more; later hits within the cache lifetime are faster and cheaper. Measure time to first token before and after enabling caching.

How does prompt caching work in Claude?

In the Claude API, you mark cache breakpoints with cache_control on content blocks, such as the end of your system prompt or a document. Claude caches everything up to each breakpoint, and later requests with the identical prefix read from the cache.

You can set up to four breakpoints, and content must exceed a minimum length to be cached. The default cache lifetime is five minutes, refreshed each time it is used, with an optional longer lifetime at a higher write cost. Responses report cache creation and cache read tokens, so you can verify hits. Cache writes cost somewhat more than normal input, while cache reads cost much less, making repeated context far cheaper.

How do you enable prompt caching in AWS Bedrock?

Amazon Bedrock supports prompt caching for supported models, including Claude and Amazon Nova. In the Converse API, add cachePoint blocks after the content to cache; with Claude's native request format, use cache_control as in Anthropic's API.

Also asked as: how to enable prompt caching in aws bedrock

Check which models and regions support caching in Bedrock's documentation, since availability expands over time. Responses include cache read and write token counts for monitoring. The same principles apply: put stable content first, keep it identical across requests, and place cache points after large static sections such as system prompts, tool definitions and reference documents. Combine caching with Bedrock's cross-region inference and batch options for cost and throughput planning.

How does prompt caching technically work?

Transformers compute key and value tensors for every input token during the prefill phase. Prompt caching stores those tensors for a prompt prefix, keyed by its exact content, so a later request with the same prefix can reuse them and only compute the new tokens.

Providers keep cached tensors in fast GPU or host memory for a limited time, evicting them when unused. Because attention for later tokens depends on everything before them, only identical prefixes can be reused; changing one early token invalidates everything after it. This is why caching works on prefixes, not arbitrary middle sections. Serving engines track cached blocks with hashes, matching new requests against stored prefixes efficiently.

Is prompt caching the same as KV caching?

They are related but different. KV caching reuses key and value tensors within a single request, so generating each new token does not recompute earlier tokens. Prompt caching persists those tensors across separate requests that share a prefix.

Every modern inference engine uses KV caching during generation; without it, producing long outputs would be extremely slow. Prompt caching extends the idea across requests, conversations or users, storing the prefill results of common prefixes for reuse. In self-hosted engines, this is called prefix caching. Both trade memory for speed, and both depend on the fact that attention for a token only looks at earlier tokens. Understanding both helps you reason about latency.

How usable is prompt caching in production?

Very usable, and often one of the easiest cost and latency wins. It shines for long system prompts, tool definitions, large documents, few-shot examples and multi-turn conversations where earlier messages repeat in every request.

Common pitfalls include dynamic content early in the prompt, such as timestamps or user names in the system prompt, which break cache hits; inconsistent tool ordering; and low traffic, where caches expire before reuse. Monitor cache hit rates through the usage fields in responses, and alert when they drop after code changes. For RAG, caching helps when the same documents are queried repeatedly, less so when every request retrieves different passages. Document caching assumptions so teammates do not break them accidentally.

How do you enable prompt caching in Claude Code?

Claude Code uses prompt caching automatically, caching its system prompt, tool definitions, project instructions and conversation history, which keeps long sessions faster and cheaper. You do not need to configure anything to benefit.

Also asked as: how to enable prompt caching in claude code

Caching is why keeping a stable CLAUDE.md file helps: unchanged instructions stay cached across turns, and smaller, focused instructions cache efficiently. Environment variables exist to disable caching for debugging, but leaving it on is recommended. Very long sessions eventually compact or clear history, which resets parts of the cache. If you build your own agents with the Claude Agent SDK, caching behaviour is handled similarly, but verify with usage data that cache reads appear as expected.

How should you structure prompts to make the most of prompt caching?

Put stable content first and changing content last: tool definitions, system instructions, reference documents and examples at the start, then conversation history, then the new user message. Keep the stable part byte-for-byte identical between requests.

Also asked as: how to structure prompts to make the most of prompt caching

Avoid timestamps, random IDs or per-user details in early sections; move them to the end. Keep tool lists in a fixed order. Place cache breakpoints after large stable sections, and in multi-turn chats, after the latest message so the growing history is reused. For RAG, put shared reference material before retrieved passages. Test by sending two identical requests and confirming the second reports cache reads rather than writes.

How does prompt caching work with paged attention and prefix caching?

PagedAttention, introduced by vLLM, stores the KV cache in fixed-size blocks, like memory pages in an operating system. Automatic prefix caching hashes these blocks so requests sharing the same prefix reuse existing blocks instead of recomputing them.

Also asked as: how prompt caching works: paged attention and prefix caching

Block-based storage reduces memory waste and lets engines share common prefixes, such as a system prompt, across many concurrent requests. SGLang's RadixAttention organises cached prefixes in a radix tree for efficient matching. Self-hosted deployments gain the same benefits as provider prompt caching: faster time to first token and higher throughput. Memory limits determine how many prefixes stay cached at once. Enable prefix caching in your serving engine's settings, and structure prompts with shared stable prefixes to maximise reuse.

What is prompt caching, and what are its best practices?

Prompt caching is a technique where a model provider or serving engine reuses the processed form of a repeated prompt prefix across requests, cutting latency and input cost. Best practices centre on stable prefixes, monitoring and matching cache lifetimes to traffic.

Also asked as: what is prompt caching? best practices explained

Order prompts with stable content first, mark breakpoints where providers require them, and avoid dynamic values early in the prompt. Monitor cache read and write tokens to measure hit rates. Choose longer cache lifetimes for content reused less often, weighing their higher write cost. Combine caching with smaller models and focused retrieval for further savings. Providers differ: OpenAI caches automatically above a size threshold, while Claude uses explicit breakpoints.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail