Skip to main content

​
Compare AI agents with percentile benchmarks

The agent comparison surface answers the question the per-agent dashboards can’t: which of my agents is actually performing, relative to the rest? It ranks every agent with traffic against each other and the org baseline on seven KPIs — cost, attributed revenue, margin, margin %, containment, resolution, and escalation — and stamps each cell with a percentile rank and quartile band, so an outlier reads as an outlier instead of just a number. Open it at Insights → Agent comparison in the Orbit console. The endpoint shapes behind it are in the Insights API reference; this guide explains how to read the dashboard, what the percentiles mean, and what to do with the ranking.

​
What cross-agent comparison does

For each selected window, the surface reuses the aggregates you already trust: cost, revenue, and margin come from the Agent ROI attribution pipeline, and containment, resolution, and escalation come from the Containment aggregates. Comparison layers ranking on top of those numbers:
  • Percentile (0–100) — where the agent sits in the org population on that metric, direction-aware so “better” always ranks higher. High containment ranks high; high escalation ranks low.
  • Rank (1 = best) — the agent’s place among the ranked agents on that metric. Hover the band badge to see “Rank X of N”.
  • Quartile band — the percentile mapped onto one of four badges: Top quartile (≥ 75), Above average (≥ 50), Below average (≥ 25), or Bottom quartile (< 25). Every agent lands in exactly one band per metric.
The org average row pins the baseline under the table — the badge-free line every agent is measured against.

​
Reading the comparison dashboard

The page has three parts, top to bottom:
  1. Date range — the same window picker as the sibling Insights dashboards (24 h, 7 d, 30 d, 90 d, 12 m, or a custom range). Percentiles are recomputed per window, so pick the window before reading anything.
  2. Agents to compare — every agent with traffic in the window, listed with its conversation count. The two busiest agents are pre-selected; you need at least two selected for the table to render. Use the search box to filter the list by name or id when the org has many agents — the picker page-izes long lists behind a “Show more” toggle.
  3. Side-by-side table — one row per selected agent (plus the org-average row), one column per KPI. Each selected agent row shows its model and conversation count alongside the seven metrics, and each metric cell shows the value plus its band badge.
Click a metric header to sort the selected agents by that metric — first click sorts best-first, second click inverts. Click the same header again to toggle back. Agents with no value on a metric (a null rate, an empty denominator) sink to the bottom of a sorted column and render the em-dash placeholder rather than a fabricated zero. The footnote under the table tells you the population the percentiles were computed against — those badges mean more when you know what “the org” means here:
  • Full population — “compares each agent against every agent with traffic in this window.”
  • Truncated population — when your org has more trafficked agents than the ranking cap, the footnote says the badges reflect the top-by-traffic subset, e.g. “the 200 highest-traffic agents (of 940 with traffic)”. The ranked subset is still the busiest subset, but don’t read it as whole-org coverage.

​
How to interpret percentiles

A percentile is a rank within the org, not an absolute judgment. Two habits keep the read honest: Per metric, direction matters. The metric catalogue knows which way is up: containment, resolution, attributed revenue, margin, and margin % rank higher-is-better; escalation and cost rank lower-is-better. So a “Top quartile” badge on escalation means low escalation, and a “Top quartile” badge on cost means low cost. Trust the badge, then confirm the raw value next to it. Sample size gates the badge. An agent with a handful of conversations can post a 100% containment rate and sit in the top quartile on a denominator of four. The conversation count next to each agent name is your volume check — treat badges on thin agents as provisional until the volume holds up. Ties share a mid-rank percentile, so two identical agents get identical badges rather than arbitrary ordering. Null values (an agent with no attributable value, or a rate over an empty denominator) carry no badge at all — the cell renders ”—”. That is “not computable here”, not zero; sorted columns push those rows to the bottom.

​
Common uses

Spot an outlier high-escalation agent. Sort by the Escalation column. An agent bottom-quartile on escalation (bad) while its containment holds top-quartile is escalating customers it could have contained — triage its guardrails and handoff rules. An agent bottom-quartile on both containment and resolution is underperforming; one with strong containment but weak resolution is looping without either a resolution or a handoff. Compare cost-per-resolution across models. The table shows each agent’s model next to its name, so you can select two agents that differ only in model and sort by cost or margin. If the cheaper model holds resolution within the same band, the margin column tells you how much the downshift is worth — feed that back into routing on the cost controls page. Catch thin-agent flukes before promoting them. A newly launched agent will often read top-quartile on a small denominator. Check the conversation count before you act on the badge — the surface always shows the volume next to the badge, so a four-conversation fluke is visible. Rebalance the fleet. Sort by margin to find the agents subsidizing the rest; sort by cost to find the agents consuming budget without outcomes. The org-average row is the bar each agent has to clear to be called “above average” on the money metrics.

​
Export and reporting

The dashboard is a read surface; for export and scheduled reporting, pull the same numbers over the API. Call GET /api/v1/insights/agent-benchmarks without agentIds to benchmark every agent with traffic, or GET /api/v1/insights/agent-comparison with a comma-separated agentIds list (2–200 ids) when you want an explicit subset plus the org-average row. Both honor the same window parameters (from / to), so a report job can snapshot the comparison your operators were reading on screen.
For full field-level detail — the metric catalogue (key, label, unit, higherIsBetter), the population-safety flags (population_size, ranked_population_size, population_truncated), and the per-agent benchmark fields (value, percentile, rank, sample_size, band) — see the Agent comparison & benchmarks section of the Insights API reference.

​
Role access

Agent comparison surfaces aggregate cost, revenue, and margin, so access matches the cost-family dashboards: the owner, admin, developer, and billing roles see the page, and API calls from other roles are rejected. Agent and viewer roles don’t see the page in the console sidebar.

​
See also