How Prompt Caching Works
What prompt caching does
Large language models process an input sequence before generating a response. That input may include system instructions, background documents, examples, conversation history, tool definitions, and a new user request. If much of this material stays the same across requests, processing it repeatedly can waste time and computing resources.
Prompt caching avoids some of that repeated work. The provider saves intermediate results for a reusable part of the input, usually an identical prefix, and uses those results when a later request contains the same prefix. The model still processes any new or changed content and generates a new response.
The important distinction is that prompt caching normally reuses computation, not an already-generated answer. It does not mean that the model remembers the conversation or automatically returns a previous response.
How prompt caching works
1. The request is divided into reusable and changing content
Consider an application that repeatedly asks questions about the same large reference document. A request might contain:
- Stable system instructions
- The same reference document
- The same examples or tool definitions
- A changing user question
The stable material can appear at the beginning of the prompt, while the changing question comes afterward. This layout gives the serving system a consistent prefix to recognize and reuse.
2. The model processes the initial prompt
When the prompt is first received, the model performs the computation needed to interpret its input. Transformer models produce intermediate key-value states during this process. These states are commonly called a KV cache or KV states.
A provider can retain suitable intermediate states associated with a prompt prefix. The exact storage, eligibility rules, and lifetime are implementation details that can differ between providers and services.
3. A later request is compared with the cached prefix
When another request arrives, the system checks whether its beginning matches a cached prompt prefix. Matching is generally based on the rendered input presented to the model, so seemingly small changes can matter. A change in an earlier part of the prompt may prevent the intended later portion from matching.
If the prefix matches, the system can reuse the saved intermediate computation. It then processes the remainder of the request and generates a fresh response.
4. A cache hit or miss occurs
A cache hit means that reusable prompt computation was found and applied. A cache miss means that no suitable cached computation was available, so the input must be processed normally. A request can be sent to a cache-enabled system without producing a hit.
Cache availability may depend on factors such as exact prefix matching, cache lifetime, capacity, request routing, and provider-specific rules. Therefore, developers should treat caching as an optimization rather than a guarantee for every request.
Why prompt layout matters
The most reliable general pattern is to put stable content first and changing content last. For example:
- Stable instructions
- Reusable reference material
- Reusable examples or tool descriptions
- Current user request
If frequently changing text is placed near the beginning, it can prevent the rest of the prompt from matching the cached prefix. Even when a large document is unchanged, a modification before the intended cache boundary can invalidate the matching suffix.
This principle applies to applications that repeatedly send long instructions, policies, product catalogs, documentation, or other shared context. The goal is not to make every request identical. It is to make the largest useful portion of the beginning identical.
What prompt caching can improve
Prompt caching primarily reduces repeated input processing. Depending on the provider and workload, that can improve:
- Latency: less repeated work may allow the request to begin generating a response sooner.
- Input cost: some services apply different accounting to cached and uncached input processing.
- Throughput: serving systems may handle repeated long prefixes more efficiently.
The exact savings, pricing treatment, and reported usage fields are provider-specific. A cache hit also does not eliminate the work required to process new input or generate the output.
Prompt caching is not conversational memory
Prompt caching and conversational memory solve different problems.
Conversational memory is an application behavior: the application stores relevant past information and includes it in a later request, or retrieves it when needed. The model only has access to information that is included in its current context or supplied through an application mechanism.
Prompt caching is an efficiency mechanism: it reuses computation for repeated input. A cached prefix may help process a repeated conversation history faster, but it does not decide what should be remembered, summarize a conversation, or create persistent semantic memory.
Prompt caching also does not expand the model's context window. The request still has to fit within the model and service limits, and the model still receives the relevant input for each request.
Prompt caching and ordinary KV caching
In transformer inference, KV caching is also used within the processing of a sequence so the system does not repeatedly recompute earlier states while generating additional tokens. Prompt caching builds on the same general idea by allowing suitable intermediate states from a reusable prompt portion to support later requests.
The terms are not always used identically across systems. “KV cache” can describe an internal mechanism used during one generation, while “prompt caching” usually emphasizes reuse across separate requests. Provider documentation may define the boundary, retention behavior, and accounting differently.
A simple practical example
Suppose a support application sends a 40-page product manual with every question. The manual and the application's instructions remain unchanged, but the customer's question changes each time.
Without prompt caching, the service repeatedly processes the manual and instructions as part of every request. With a matching cached prefix, the service may reuse the intermediate computation for that stable material, then process the new question and produce a new answer. If the application inserts a changing timestamp or request identifier near the beginning, the intended prefix may no longer match.
Limitations and privacy considerations
Prompt caching is not guaranteed. A cache may expire, be unavailable because of capacity or routing, or fail to match because the input changed. Different providers may also use different matching rules and retention policies.
Developers should understand what content may be retained by the service, how long it may remain eligible for reuse, and whether the provider documents isolation or privacy behavior for cached prompts. Sensitive information should not be placed in a prompt merely because caching might improve performance. Privacy and data-handling decisions depend on the provider, deployment, configuration, and applicable policies.
Common misconceptions
- “The answer is cached.” Usually, prompt caching reuses input processing, not the generated answer.
- “Every cached request is a cache hit.” A cache-enabled request can still miss.
- “The model remembers the prompt permanently.” Cache lifetimes and availability are implementation details, not persistent memory.
- “Changing the last sentence breaks the entire cache.” A stable prefix can still be reused when changes occur after the cacheable boundary.
- “Prompt caching makes the context window larger.” It reduces repeated computation but does not change how much input the model can accept.
How to use prompt caching effectively
When designing an application, identify the material that is repeated across requests and place it before request-specific content. Keep the reusable prefix stable, avoid unnecessary changes near its beginning, and monitor whether requests actually produce cache hits.
Finally, evaluate caching using the measures that matter for the application: response latency, input-processing cost, throughput, and correctness. Prompt caching should complement good prompt and application design; it should not be treated as a substitute for managing context, memory, or retrieval.