Your Eval Set Is Lying to You

Teams often reach for two quick evaluation approaches because they're easy to stand up without labeled data. Both have been tested and found brittle.

• Judge panels (LLM voting): ask several models to judge whether an output is correct and take the majority vote. This requires no labels but can give a false sense of security — panel agreement does not imply correctness in real traffic.

• Step-level grading: decompose an agent's behavior into steps, grade each step, and blame the first failing step. This seems intuitive but can miss causal failure modes because it treats observed trajectories as if errors are local and isolated.

Recent studies replaced observational checks with causal interventions: researchers made a known change in the system at a known point, then checked whether the evaluation signal detected that change. Both judge panels and step-level grading failed to reliably detect the induced changes. That suggests these cheap signals can be blind to important faults and should not be the only source of truth.

Practical steps to improve your evals

• Align to traffic: build eval slices that mirror the real inputs, edge cases, label distributions, and downstream success criteria you see in production.

• Validate signals with interventions: introduce small, controlled changes you understand and check whether your eval detects them. If it doesn't, the eval may be insensitive to important failures.

• Avoid over-reliance on model judges: panels that agree frequently can still be systematically wrong. Treat majority votes as a noisy signal, not ground truth.

• Treat step-level grading cautiously: step decomposition can help debugging, but don't assume step scores imply final outcome. Test whether step failures correlate with end-to-end degradation.

• Continuous monitoring and slices: collect production telemetry and build targeted evals for slices where errors matter most. Use production feedback loops to update the eval set.

• Combine signals: use labeled examples for critical paths, synthetic interventions for fault sensitivity checks, and human review on high-risk outputs.

If your model performs well on an eval but poorly in production, start by interrogating the eval. Cheap, fast signals can help triage, but they do not replace application-aligned evals and causal validation. Design evals that match your real traffic, validate them with interventions, and keep monitoring in production.