Dual-layer monitoring for production agents: AgentCore Evaluations plus AWS DevOps Agent

# Why traditional monitoring misses agent failures

Multi-agent production systems can fail in ways that conventional infrastructure monitoring does not catch. An agent can invoke Bedrock successfully, call tools without obvious errors, and still fail to accomplish the user's goal. Problems often show up as reduced or incorrect behavior rather than 500-level errors. In multi-agent flows, a supervisor may reroute a portion of requests to the wrong specialist or a booking agent may stop completing reservations while logs show successful tool calls three calls deep.

# The two monitoring questions

Operational monitoring needs to answer two separate questions:

• Is the agent doing the right thing for the user? (quality)
• Is the underlying infrastructure supporting correct agent behavior? (health)

Treating these as distinct concerns avoids blind spots where infrastructure metrics look healthy but agent outputs are wrong, or vice versa.

# Layer 1: AgentCore Evaluations — continuous quality monitoring

AgentCore Evaluations is a quality-assessment layer integrated with the AgentCore runtime. It samples production interactions and scores them on dimensions such as helpfulness, correctness, and goal completion. Each score includes reasoning tied to the conversation, tools used, and task requirements. When scores fall, the system runs pattern analysis across the recent low-scoring sessions to identify common failure modes, such as wrong tool selections or routing mistakes.

• Background evaluation of a configurable sample of live requests.
• LLM-as-a-judge approach to generate scores and textual reasoning for each evaluation.
• Pattern analysis across sessions to surface recurring quality regressions.

This layer tells you whether the agent is actually helping users achieve the task, which CloudWatch metrics do not capture.

# Layer 2: AWS DevOps Agent — autonomous infrastructure investigation

AWS DevOps Agent acts like an on-call engineer for infrastructure incidents. When an incident is detected, it correlates CloudWatch logs, traces, IAM policies, and orchestration telemetry to trace failures across service boundaries. It offers root-cause analysis and remediation recommendations without manual investigation.

• Infrastructure problems can be silent: revoked permissions or throttling may prevent downstream behavior without raising errors that appear in basic logs.
• Multi-agent routing and long chains of tool calls create tracing blind spots unless you correlate across traces, logs, and IAM contexts.

AWS DevOps Agent fills that gap by autonomously following the signals and mapping them to likely causes.

# together

In an example airline reservation system built on AgentCore runtime, the two layers provide complementary coverage:

• AgentCore Evaluations detects that bookings have dropped or that a supervisor started sending 20% of requests to the wrong specialist.
• AWS DevOps Agent investigates and finds an IAM permission change or a throttling event that explains the behavior.

AgentCore Evaluations answers the "did we help the user?" question. AWS DevOps Agent answers the "why did the system change its behavior?" question. Both are necessary in a swarm-style, dynamically routed multi-agent architecture.

# Practical implications for teams

• Instrument agents with OpenTelemetry so AgentCore runtime can emit the traces and context required for both evaluation and investigation.
• Configure sampling rates for AgentCore Evaluations to balance cost with the need for representative signals.
• Use DevOps Agent to reduce time-to-root-cause across service boundaries, especially for issues that don't surface as classic errors.

# Final point

Monitoring multi-agent systems requires explicitly tracking agent quality alongside traditional infrastructure health. Combining continuous evals that judge agent responses with autonomous infrastructure tracing provides the two views needed to detect, diagnose, and act on failures that would otherwise remain hidden.