A practical walkthrough that shows how to build a provider-neutral Python harness that varies presentation while holding task, evidence, and policy constant, so you can detect when context changes alter decisions.
Define the policy and expected outcomes before creating prompt or evidence variants so graders can’t silently reinterpret unknowns.
Generate controlled prompt and evidence variants (including evidence-order changes) and preserve failing attempts and configuration for postmortem analysis.
Validate the harness locally with passing and deliberately defective fixtures before connecting a live model or application through a read-only adapter.
This tutorial builds a controlled evaluation workflow in Python that tests whether an AI reviewer preserves and applies a declared policy across presentation variations. The goal is to detect context-sensitivity bugs: cases where the same substantive evidence leads to different decisions because of prompt wording, evidence order, or execution conditions.
Start by writing the requirement as a deterministic rule with explicit outcomes. The article's worked example is a two-node power-domain requirement with three cases and expected decisions:
Treat unknown as an explicit value (Python None / JSON null) so graders don't invent interpretations. Include all outcomes so a grader that always approves or always holds will fail some cases.
Keep the answer key outside the application
Create variants that change only presentation elements: equivalent prompts, different orders of evidence, and alternate instruction phrasing. Do not change the task, the evidence values, or the policy while varying presentation. Serialize None as JSON null when sending prompts so the application sees the explicit unknown.
Before connecting a live model, run deterministic fixtures locally. Include a passing fixture (the expected behavior) and a deliberately defective fixture to ensure the harness detects regressions and known failures. Use only the standard library in the supplied reference implementation so it can run without external dependencies.
The harness defines an adapter interface for provider-specific integrations. When ready to test live systems, connect an approved application in read-only mode. Keep deployment credentials and mutation-capable tools out of the evaluation environment.
Preserve failures and analyze by condition
Record all failed attempts and configuration used for each trial. Analyze results by condition (prompt variant, evidence order, policy case) instead of relying only on aggregate scores. Use the suite as a regression check that flags when behavior changes under controlled variations, not as an authority that certifies model reliability.
The example harness focuses on a single, advisory decision and synthetic fixtures. It is a reference implementation for local demonstration and regression testing, not a comprehensive architecture review, retrieval benchmark, or production assurance product.
A defensible context-sensitivity harness requires: explicit policy definitions, externalized answer keys, controlled presentation variants, independent graders, preserved failures, and upfront validation. Treated as a regression tool, the harness makes prompt changes testable rather than anecdotal.
You changed a prompt and the outputs look different. Better or worse? Without a way to check, that question gets answered by whoever argues hardest, and this builds the thing that answers it instead.
When I first started building enterprise applications with Large Language Models (LLMs), I fell into a trap that almost every developer encounters. I thought that scaling an AI system simply meant refining a single, massive prompt. I wrote complex system instructions, packed the context window with rules, and expected
An AI model wrote 69 tests for a Python module. Every one passed. Together they caught zero of the eleven bugs I had deliberately planted in that module. A second setup, pointed at the specific bugs rather than at the module, used 17 attempts and caught ten. That contrast is the whole project. What I built A mutation-t
The race to build better AI software has largely focused on the model: which one reasons better? Codes better? Costs less? At Adronite, our testing points to another major efficiency lever that gets far less attention: the harness surrounding the model. Our internal benchmarking suggests the choice of harness can mater
Loading more related stories...
Open the app view to save this story, compare related coverage, and continue from the same source.