Prefix Caching

In depth

An inference-engine optimization that keeps the KV cache from earlier requests and reuses it when a new request starts with the same tokens, so the shared prefix skips prefill. In vLLM each full cache block is hashed from its tokens plus its parent block's hash, so a block only matches when everything before it matches too, and unused blocks are evicted least recently used first. Provider prompt caching applies the same idea as a billed API feature, and both reward putting static content like system prompts and tool definitions at the start of the prompt.

Primary source: docs.vllm.ai