Agent Evaluation Metric (AEM): A turn-level way to find the single turn that causes multi-turn failures

# The correctness problem in multi-turn agentic conversations

Multi-turn agents fail in ways single-turn or outcome-level evaluation misses because one early error often propagates across later turns. A task-level check that only inspects the final outcome marks the whole interaction as failed without indicating which turn actually broke. That makes debugging and targeted fixes expensive and slow.

# What AEM (Agent Evaluation Metric) does

# Correctness: the first AEM dimension

In this post correctness is the first dimension AEM instantiates. Correctness itself is split into two concrete sub-metrics:

• Truthfulness: Are the values the agent produced factually consistent with what was expected? This covers both parameter values in tool calls and factual statements in natural-language responses.
• Completeness: Are all required elements present? This checks for missing parameters, omitted required fields, or partial responses that don't include requested information.

# Turn-level hierarchy: response turns and action turns

# Why this matters operationally

• Faster debugging: Replace guessing at which turn failed with a specific turn-level failure signal.
• Focused fixes: Teams can correct the root cause (for example a wrong tool parameter) rather than reworking downstream logic that only reflected the original error.
• Extensible metrics: The same decompose-evaluate-compose pattern can add new quality dimensions (safety, instruction retention, reasoning depth) without redesigning the metric machinery.

# Failure taxonomy and actionable scores

AEM pairs per-turn scores with a failure taxonomy that makes results actionable. Rather than returning only a pass/fail at the task level, AEM returns named sub-metric states per turn. That lets you track whether an interaction failed because it was untruthful, incomplete, or both, and whether the problematic turn was a response or an action.

# How teams can start using the idea

Start by instrumenting logs to record turns as either response or action, capture tool call parameters and returned values, and apply simple truthfulness and completeness checks per turn. Use the per-turn signals to locate the first failing turn in a trajectory and prioritize fixes that address that turn's sub-metric failures.

# Bottom line