As the model processes each token, its attention layers create key and value representations for that token. When the next token arrives, the model uses those stored key and value representations instead of recalculating them for all prior tokens. That stored attention state is the KV cache.
KV cache is not the source document, chat transcript, RAG index, or agent plan itself. Those remain application data. KV cache is the model’s internal working state for tokens it has already processed.
Every retained token consumes KV cache memory. The total requirement grows with:
Long documents, larger RAG inputs, tool-heavy agents, extensive code context, and extended project sessions can all create more active KV cache demand than available GPU or unified memory can hold.
A live workload needs more retained KV cache capacity than fast memory can hold.
Related tasks repeatedly prefill the same unchanged input.
Challenge | What it means |
|---|---|
KV cache extension | A live workload needs more retained KV cache capacity than fast memory can hold. |
KV cache reuse | Related tasks repeatedly prefill the same unchanged input. |
Pascari aiDAPTIV™ helps supported workloads retain eligible less-active KV cache state across GPU memory, system memory, and cache memory instead of discarding it immediately when fast-memory capacity is constrained.
GPU memory or unified memory remains the primary working-memory. Lower tiers add retained capacity; they do not make flash perform like GPU memory.
KV cache reuse depends on token identity.
The shared token sequence must match from the beginning of the prompt through the portion being reused. The first mismatch ends reuse for the remainder of that sequence.
Two prompts can be semantically similar but still fail to reuse cache if their token order, punctuation, formatting, tool definitions, document order, or system prompts differ.
That makes prompt structure important. A stable system prompt, fixed tool definitions, a long source document, fixed context blocks, or a repeatable agent configuration can create a useful reusable prefix.
When a reusable prefix matches, the system can avoid recomputing that processed portion of context.
The new question, changed suffix, tool response, or next agent step still needs to be processed.
Reuse depends on runtime support and compatible input. Applications must preserve shared material in a stable order, with changing input appended later.
To maximize reuse, keep shared instructions, tools, and documents at the beginning and place changing input (questions, tasks, or steps) at the end.
Traditional RAG can limit reuse when every query assembles retrieved chunks in a different order.
A cache-aware RAG workflow organizes related material into fixed context blocks. It can retain the KV cache state for each block, retrieve relevant material for a new question, select a compatible block as the stable prefix, and append the live question afterward.
This does not eliminate retrieval or reasoning. It reduces repeat work when related questions depend on the same underlying material.
Still retrieves and reasons; reduces repeat work.
An agent often works through a loop:
Each step can add system instructions, tool schemas, search results, database responses, retrieved documents, intermediate outputs, working artifacts, and outputs from sub-agents.
A task that begins with a modest context window can become a long-running memory problem before you see the final result.
KV cache extension and reuse are most useful when meaningful context remains stable across related work:
It is less useful when every prompt changes from the first token onward, the shared prefix is short, context is rarely reused, or the runtime does not support the required cache behavior.
Learn how aiDAPTIV can extend eligible KV cache retention and reuse compatible shared input across document, RAG, coding, and agent workflows.
English English 
繁體中文
简体中文
日本語
