What Is a KV Cache?
A KV Cache stores previously computed attention data so transformer models can generate new tokens without recalculating the entire conversation.
Definition
A KV Cache (short for Key-Value Cache) is a memory optimization technique used by transformer-based AI models to speed up text generation. It stores previously computed key and value representations from the model’s attention mechanism so they can be reused instead of recalculated every time a new token is generated. KV caching belongs to the fields of transformer architecture, inference optimization, and large language model (LLM) engineering.
KV Cache matters because it allows modern language models to generate long responses efficiently. Without it, the model would repeatedly recompute information for the entire conversation after every new token, making inference dramatically slower and more computationally expensive.
In One Sentence
A KV Cache stores previously computed attention data so transformer models can generate new tokens without recalculating the entire conversation.
Key Takeaways
A KV Cache speeds up inference by reusing previously computed attention information.
It stores the keys and values created by each transformer layer.
KV Cache improves generation speed but increases memory usage.
Longer conversations require larger KV caches.
Nearly all modern transformer-based language models rely on KV caching during inference.
Why KV Cache Matters
Most people using AI never see the KV Cache, yet it is one of the main reasons modern language models feel responsive.
Imagine asking a chatbot a long series of questions. Without a KV Cache, the model would need to reread and recompute the entire conversation every time it generated the next word. Even a modest conversation could become painfully slow.
Instead, the model remembers the expensive calculations it has already performed.
For people running local AI models, KV Cache is particularly important because it directly affects memory usage. When a conversation becomes longer, the model itself does not grow larger—but its KV Cache does. This is why applications like LM Studio, Ollama, llama.cpp, and similar inference engines often display both model memory and KV Cache memory separately.
Understanding KV Cache helps explain why context length, RAM, and VRAM requirements increase during long conversations even when the model itself remains unchanged.
How KV Cache Works
To understand the KV Cache, it helps to know a little about how transformer models process text.
Transformer models use a mechanism called self-attention to determine which earlier words are relevant when generating the next token.
For every token, the model creates three mathematical representations:
Query (Q)
Key (K)
Value (V)
The current token’s query is compared with the keys of previous tokens to determine which parts of the context deserve attention.
Once the model has identified the relevant tokens, it retrieves the associated values, combining them to produce the next prediction.
Computing these keys and values is computationally expensive.
Fortunately, once a token has been processed, its keys and values never change.
This makes caching possible.
Imagine writing a book.
Instead of rereading every previous chapter whenever you write the next sentence, you keep detailed notes summarizing everything already written.
Each new sentence only requires consulting those notes and adding one more.
The KV Cache works in much the same way.
Suppose a conversation begins:
User: What is machine learning?
The model computes keys and values for every token in that prompt and stores them in the KV Cache.
Later the user asks:
How is it different from deep learning?
Instead of recomputing the keys and values for the first question, the model simply reuses the cached data and computes new keys and values only for the newly added tokens.
As the conversation continues, the cache gradually grows.
This dramatically reduces the amount of computation needed during inference.
The trade-off is memory.
Every additional token requires storing more keys and values.
For long contexts—32,000, 64,000, or even hundreds of thousands of tokens—the KV Cache can consume several gigabytes of RAM or VRAM.
The size of the KV Cache depends on several factors, including:
the number of tokens in the context;
the number of transformer layers;
the hidden dimensions of the model;
the numerical precision used for storage.
Larger models generally require larger KV caches because they contain more layers and larger internal representations.
Some modern inference engines support KV Cache quantization, which stores the cached keys and values using lower numerical precision.
This reduces memory usage while usually having only a small effect on generation quality.
KV Cache is used during inference, not training.
During training, models process many sequences simultaneously and continuously update their parameters, so caching previous attention states generally does not provide the same benefit.
Common Misconceptions About KV Cache
Misconception: The KV Cache stores the conversation text.
Not exactly. The conversation text is stored separately. The KV Cache stores mathematical representations of that text that the attention mechanism can reuse efficiently.
Misconception: Clearing the KV Cache deletes the model.
It does not. Clearing the cache simply removes temporary inference data. The model weights remain unchanged.
Misconception: KV Cache improves the model’s intelligence.
No. KV Cache only improves efficiency. It makes generation faster but does not increase the model’s reasoning ability or knowledge.
Misconception: Quantizing the model automatically reduces KV Cache memory.
Not always. Model quantization reduces the size of the model weights. Unless the inference engine also supports KV Cache quantization, the cache may still use higher numerical precision.
Comparing KV Cache with Similar Concepts
KV Cache is often confused with the context window, but they are different concepts.
The context window defines how many tokens a model can consider at once.
The KV Cache stores the internal attention data corresponding to those tokens so they do not need to be recomputed during inference.
KV Cache also differs from model weights.
Model weights are the learned parameters stored permanently after training.
The KV Cache contains temporary data created during a conversation and is discarded when the session ends or the cache is cleared.
Finally, KV Cache should not be confused with system memory caching performed by an operating system.
Operating system caches speed up file access or disk operations.
KV Cache is a specialized optimization inside transformer inference that accelerates attention calculations.
See Also
Transformer
KV Cache exists because of the transformer architecture. Understanding transformers provides the foundation for understanding why keys and values are generated in the first place.
Self-Attention
The KV Cache stores data produced by the self-attention mechanism. Learning how self-attention works makes the purpose of the cache much clearer.
Context Window
The context window determines how much text a model can consider at once. Together with KV Cache, it explains memory usage during long conversations.
Inference
KV Cache is used during inference to speed up text generation. Exploring inference helps explain where KV caching fits into the overall AI workflow.
Token
Every new token added to a conversation expands the KV Cache. Understanding tokens helps explain why cache size grows over time.
Quantization
Some inference engines support KV Cache quantization to reduce memory usage. Learning about quantization provides useful context for these optimizations.
Large Language Model (LLM)
Most modern LLMs use KV Cache during inference. Understanding LLMs shows how KV caching contributes to responsive conversational AI.
VRAM
KV Cache often occupies a significant portion of GPU memory. Understanding VRAM helps explain the hardware requirements of local AI models.
Prefill
Before token-by-token generation begins, the model performs a prefill stage that computes the initial KV Cache for the input prompt. Understanding prefill completes the picture of how transformer inference works.