Nvidia finds that simple linear math can replace costly AI model handoffs

The useful part

This is a major bottleneck for enterprises building long-horizon, multi-LLM workflows. This technique aligns with real-world agentic applications where large contexts accumulate across many turns. For real-world AI applications, cross-model KV cache transfer can reduce compute costs and latency on long-running, multi-LLM workflows — and it does so with simple linear math, not an expensive deep learning model.

How it works

• Experiments show that, on compatible model pairs, this linear mapping process runs 2.7 to 25 times faster than recomputing the conversation while retaining up to 98% of the target model's standalone accuracy.
• Why swapping models mid-session is so expensive Examining how LLMs handle memory helps understand why multi-model workflows hit a performance wall in production.
• When an LLM receives a prompt, it must first execute the "prefill" stage, which is the initial forward pass that computes the keys and values for all input tokens and populates the Key-Value (KV) cache.
• After that, it enters the "decode" phase, where it computes and generates the next tokens in the sequence.
• In multi-turn conversations or long-horizon agentic sessions, the context gradually becomes longer.

What to take from it

Because the computational cost of the prefill stage scales directly with both model size and input length,...