Evaluation-First AI Agents: How Zepto Uses Databricks and MLflow to Scale Customer Support

Why Zepto changed how it builds agents

Zepto runs customer support on an AI multi-agent system that handles over 100,000 tickets per day. Early iterations focused on shipping agents quickly, but scale exposed new failure modes: seasonal spikes, new product categories, multilingual requests, and multi-step workflow failures that were invisible until customers complained. A small error rate at that volume translated into many bad outcomes and revenue leakage.

Instead of prioritizing speed-to-production, Zepto made evaluation the primary way agents are built and operated. That means designing tests and baselines that exercise multi-step agent behavior (intent classification, knowledge retrieval, tool calls, transactional steps, and response generation) and gating releases on evidence that a candidate version beats a production baseline.

Databricks and MLflow serve as the infrastructure to store traces, run evaluations, and version baselines. Key components include:

• Traces: detailed logs of tool calls and intermediate steps so failures are visible rather than hidden behind a final answer.
• Golden datasets: curated examples used as baselines for regression testing and metric calculations.
• LLM-as-judge evaluations: automated judges that rate outputs against baselines to speed large-scale evaluation.

Dual-loop architecture and quality gate

Zepto organizes agent development and operations into two loops:

• Development loop: teams design, iterate, and evaluate agent versions against baselines in Databricks/MLflow until they meet reliability criteria.

• Quality gate: a strict decision point that only lets agent versions into production if they meet agreed thresholds across reliability and stakeholder metrics.

Concrete outcomes Zepto reports include:

• Operational scale: 80%+ of tickets fully managed by AI agents with human oversight available.
• Cost: 65% reduction in support cost or support tickets and a payback period of less than one month.
• Quality: 20% improvement in customer satisfaction (CSAT) and 8% improvement in accuracy.
• Velocity: 3x faster development cycles and 4x faster time to resolution.

Beyond metrics: governance and trade-offs

The evaluation-first approach also improved governance. Versioned evaluations and audit trails make deployment and rollback decisions traceable. The same evaluation data lets teams compare models, prompt strategies, or hybrid routing strategies on concrete cost-vs-quality trade-offs rather than gut feeling. Stakeholders can capture different success criteria and make trade-offs explicit and measurable.

• Visibility matters: tracing every tool call prevents silent failures and uncovers wasted spend.

Zepto's approach reframes agent engineering as an evaluation-first discipline supported by Databricks and MLflow. The architecture focuses on observability, automated regression testing, and a hard quality gate so agents can scale reliably while controlling cost and risk.