Prefix caching (Automatic Prefix Caching, APC)
Reusing the cached KV of shared prefixes (system prompts, RAG context, chat history) to skip redundant prefill.
Reusing the cached KV of shared prefixes (system prompts, RAG context, chat history) to skip redundant prefill. Big TTFT win for shared-prefix workloads.