Prefix-aware routing for SageMaker Inference reduces LLM time-to-first-token and raises KV cache hit rates

When applications use large language models, prompts typically have a fixed prefix (system instructions, reference docs, conversation history) followed by a short variable user input. Many LLM-serving frameworks cache the model's computed key-value (KV) pairs for those repeated prefixes so subsequent requests skip recomputing that work. That technique—prefix caching—cuts time-to-first-token (TTFT) because the model only needs to process new tokens.

• Overload protection: if the instance handling a popular prefix is at its configured concurrency limit, the endpoint routes excess requests to less busy instances to avoid overload. Those requests may miss a cache hit but avoid service degradation.
• Stable scaling behavior: when instances are added or removed, most requests continue to hit the same instance as before. Only a small fraction shift, so caches don't get invalidated broadly during scale events.

Benchmarks used Llama 3.1 70B with vLLM (prefix caching enabled) on seven ml.p5.48xlarge instances. Tests compared prefix-aware routing to default random routing across multiple endpoint configurations.

Long-context workloads (8,000-token shared prefixes, sustained 1 hour):

• P50 TTFT reduced by 71–77%.
• P90 TTFT reduced by 33–37%.

• Throughput increased 15–16%.

Short-context workloads (ShareGPT-style variable conversations, 30 minutes):

• P50 TTFT reduced by 13–16%.
• P90 TTFT reduced by 24–37%.

• Throughput increased 1.7–2.0%.

Interpretation: the longer the shared prefix, the greater the benefit, because more computation is skipped on each cache hit.

Operational impact and cost considerations Routing adds 1.3–1.9 milliseconds per request. In the tests, model TTFT ranged 63–280 ms, so the routing overhead was small relative to savings. Traffic distribution remained balanced: each of the seven instances received about 13.3–15.4% of requests, within 1% of an even split, and no hot spots appeared.

Choose prefix-aware routing when your workload repeatedly sends identical or near-identical prompt prefixes and you host a multi-instance endpoint where prefix caching is supported. It delivers the largest latency and throughput gains for long-context prompts where the prefix is substantial.

If requests are interchangeable, prompts have minimal shared context, or you run non-LLM models, the default RANDOM strategy remains appropriate. SageMaker Inference continues to offer RANDOM and LEAST_OUTSTANDING_REQUESTS alongside the new option.

• Identify endpoints that repeatedly include long, fixed prefixes in requests.
• Enable prefix-aware routing and configure concurrency limits to protect instances.
• Monitor KV cache hit rates, TTFT, and instance-level request distribution during scale events.

Prefix-aware routing provides a straightforward network-layer way to make prefix caching effective at fleet scale without client-side affinity logic or tagging.