Gorgias, a ~$100M ARR ecommerce CX vendor, published a continuous, open-sourced evaluation of AI agents using 8,356 live conversations across 18 vendors and 212+ storefronts. The dataset, test harness, and weekly results reveal where Gorgias wins, where it loses, and why transparent, repeatable testing matters for buyer decisions.
Transparent, continuous tests of live AI agents give buyers checkable evidence that benchmarks and feature claims cannot.
Gorgias published an open test harness and weekly results: 8,356 live conversations, 18 vendors, 212+ stores, with each 10-turn conversation run against every vendor.
The report shows trade-offs: Gorgias leads on answer quality and overall support composite, while competitors lead on automation rate and pre-sale speed (Envive beats Gorgias on pre-sale speed).
# Why publish deep, direct competitive evals?
Vendors routinely claim superiority, but claims rarely come with raw, repeatable evidence. Gorgias published a different kind of benchmark for ecommerce CX: continuous, live-agent tests with the code and data available for anyone to check. That makes the results actionable for buyers who otherwise juggle demos, surveys, and vendor slides.
# What Gorgias published
Gorgias ran the same 10-turn conversations against every vendor's live agent, using the same chat widget shoppers use. The key facts they released:
Every conversation was scored for resolution, correctness of the answer, and latency. The tests run daily and publish weekly so improvements or regressions show up quickly.
# Concrete findings and trade-offs
The report doesn't cherry-pick only wins. It shows where Gorgias leads and where competitors do better:
Gorgias chose weightings that emphasize real shopper behavior. For example, pre-sale scoring gives speed 25% of the score while the support composite weights speed at 10%. Using the support weights for pre-sale would move Gorgias to first place (74.3), showing how weight choices change rankings.
# Why this matters more for AI agents
Traditional software behaves the same when you click the same button. AI agents do not. An agent can answer well one day and degrade after a model update. Static checklists, demo slides, or annual benchmarks miss that variability. Continuous, live testing captures real-world volatility and any improvements vendors ship.
# Buyer and vendor implications
For buyers: a published, open evaluation does much of the diligence that most brands won't do themselves. Running thousands of conversations across many vendors is impractical for most procurement teams. A transparent benchmark with raw data gives a faster, verifiable shortcut to shortlist decisions.
For vendors: publishing honest results can build credibility. Showing competitors ahead in specific metrics—while making the harness and rubric public—makes positive claims more believable. It also forces vendors to keep iterating, because results update weekly.
# Practical next steps for vendors considering public evals
# Bottom line
ICONIQ just published The Pacesetter Index. It replaces their Enterprise Five Scorecard, and shows the real growth rates and metrics for the top venture-backed startups. Not the average. But to be clear, what most VCs want to fund, The pool: the top public software companies plus ICONIQ’s own private venture and growth
Most tools for SaaS competitive analysis do one job well and three jobs badly. I found that out the slow way while building ImagineLab.art, our AI creative platform. I wanted one dashboard that showed me what rivals shipped, what they ranked for, and where their traffic came from. It does not exist. So I run […] The po
Every product roadmap conversation with a technology vendor now runs through one filter that has nothing to do with the roadmap itself: does the company have an AI story that a buyer's finance team would actually underwrite. Forrester Research just gave that gut check a formal structure. Its newly introduced AI Disrupt
Unlock B2B SaaS visibility. Discover what AI recommendations reveal about how companies get cited, from citation sources to content format and freshness signals.
Here is what SaaStr AI did for Replit over the past 12 months, without being asked, without being an investor at SaaStr Fund (oh well): Mentioned them in 214 articles Mentioned them in 40+ podcasts reaching hundreds of thousands Generated 5,901,900+ page views and impressions for them, across an audience of roughly 450
AI (Artificial Intelligence) နည်းပညာက အá€á€¯á€¡á€á€»á€á€”်မှာ နေရာá€á€á€¯á€„်းမှာ ရှá€á€”ေပါပြီዠဒါပေမဲ့ “AI ကá€á€¯ ဘယ်ကနေ စလေ့လာရမလဲአအမြန်ဆုံး á€á€á€ºá€™á€¼á€±á€¬á€€á€ºá€¡á€±á€¬á€„်â
Loading more related stories...
Open the app view to save this story, compare related coverage, and continue from the same source.