Reusing the Prompt Prefix with a Key-Value Cache for SLM Optimization

# Why cache the prompt prefix Narrow automation prompts tend to be mostly static: an instruction block, a taxonomy, and a few examples. Only a short tail changes per item. Transformers build a key and a value vector for each token at every layer, and those vectors depend only on tokens to the left. If the prefix is fixed across calls, those vectors are identical each time. Computing them once and reusing them reduces per-item work to the few tokens that actually vary.

# Baseline and environment The article uses Qwen2.5-0.5B-Instruct in float16 via Hugging Face Transformers. Benchmarks run on an M2 MacBook Air with 24GB RAM and a 16-core Neural Engine. The required Python packages shown are torch, transformers, and accelerate (pip install torch transformers accelerate). A short, realistic few-shot prompt is re-encoded for every ticket in the baseline.

# Constrained scoring to minimize work To keep the evaluation cheap, the author carries forward a constrained scoring trick introduced earlier: if label first tokens are distinct, comparing logits for each label's first token is enough to choose the label. That turns the decision into a single forward pass after the prompt prefill is handled.

# Token-clean prefix/suffix split The chat prompt uses a ChatML-style layout written out by hand so the script can split the prompt at a known boundary. The split must be token-clean: encoding the two halves separately must give the same token ids as encoding the whole prompt in one go. If that check fails, cached keys will not align with the model's expectations.

# Concrete elements included in the walkthrough

# What caching changes in practice Instead of re-encoding and recomputing keys/values for every token of the entire prompt for each ticket, cache the per-layer key and value tensors for the fixed prefix. For each new ticket you only encode the ticket-specific tail, run the smaller per-item prefill and the constrained scoring forward pass, and pick the label by comparing logits for the stored label-first-token ids.

# Implementation details to watch for

• Tokenization must be identical between cached-prefill generation and later inference calls. - The chat template layout matters: writing the template by hand helps control the token boundary. - Label-first-token uniqueness is required for the constrained-scoring shortcut.

# Practical outcome When your automation prompt is largely static and the per-item change is a short tail, caching prefix key/value tensors cuts the repeated computation substantially. The article illustrates this with code and a working validation that the prefix/suffix split is token-clean, using Qwen2.5-0.5B-Instruct as the reference model.

# Next steps suggested by the series