Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints

AWS What's New ยท 2026-11-16

Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints, enabling you to reduce time to first token for LLM workloads with long or repeated prompts. Serving engines such as vLLM store the computed key-value (KV) cache for repeated prompt starts such as shared system prompts, documents, and conversation history, to skip repeated work. Across several instances, a request that lands where the cache is missing recomputes it. Under high load... (excerpt trimmed)

Source