Evaluate what the agent actually did and whether the service is restored, not whether its explanation sounds plausible. Separate conversation, operational evidence, and outcome into distinct checks and grade each with explicit verdicts.
Collect evidence outside the agent’s control and bind observations to trial, incident, tenant, resource, and action.
Use an offline evidence grader that returns PASS / FAIL / INCONCLUSIVE and records reasons for investigation.
Treat verification as a release gate: measure repeatability, preserve known failures, and troubleshoot measurements before changing models.
# What this is about AI agent evaluation should measure the agent's observable behavior and the resulting service state, not rely on the agent's explanation. The article provides a practical contract and an example offline Python grader that distinguishes control violations, inappropriate decisions, missing evidence, and true task outcomes.
# Core ideas Evaluation must preserve four separate threads: the conversation (what the agent claims), the execution receipts (what tools attempted and executed), the service checks (whether the target operation actually works), and ticketing records (whether closure or escalation occurred). Combining these into a single quality score hides meaningful failures.
The grader model in the article returns three verdicts—PASS, FAIL, INCONCLUSIVE—plus reasons an engineer can act on. It is designed to be used by a trusted evaluation harness that supplies observations. The agent must not be able to control or fabricate the evidence path.
# Concrete checks to run
A favorable final-state check does not cancel a control violation. Missing evidence must not be treated as a pass merely because the agent's explanation appears correct.
# Implementation pattern
# Operational guidance
# Bottom line A passing explanation is not a substitute for a passing workflow. Build evidence pathways that the agent cannot manipulate, grade execution, behavior, and outcomes separately, and use explicit verdicts so engineers can investigate and fix the right problems.
<figure data-wp-context="{"imageId":"6aa4e934c328a"}" data-wp-interactive="core/image" data-wp-key="6aa4e934c328a" class="wp-block-image size-large wp-lightbox-container"><img data-recalc-dims="1" decoding="async" width="900" height="506" data-attachment-id="15130" data-permalink="https://digitalthoughtdisruption.com/2
LLM evals score a model. End-to-end agent testing gates a build. See what each one proves, where they disagree, and how to run both without duplicating work.
![Evaluating Agents Against What They Are Actually Supposed to Do [Testμ 2026]](/api/proxy/image?url=https%3A%2F%2Fassets.testmuai.com%2Fresources%2Fimages%2Fmeta%2Fevaluating-agents-against-intent.webp)
Francesca Lazzeri of Microsoft on why generic agent metrics miss real failures, and the four-layer evaluation loop built on ASSERT, an open-source framework.
Agentic AI architecture splits planner, generator, and evaluator roles. Why a generator cannot grade its own output, and how to wire an independent evaluator.
Our app worked. That was the problem. My team had spent roughly half a year working with our developer and using AI to build an in-house learning management app. In our testing environment, every function appeared to work. Zoom links could be updated. Calendars were connected. The automated checks reported that the sys
Most people who work in AI spend a lot of time explaining AI. They build slide decks. They screenshot dashboards. They create product walkthroughs with voiceovers and screen recordings. They talk about capabilities, use cases, potential ROI. And most of that explaining… does very little. Here’s the approach that actual
Loading more related stories...
Open the app view to save this story, compare related coverage, and continue from the same source.