Agent custom outcome metrics

Every AI agent in Orbit has an Outcomes tab where you define the business results you care about — resolved ticket, booked appointment, correct escalation — as rubrics. An LLM judge reads each closed conversation against every rubric and records a pass/fail row with a confidence score and a line of reasoning. Those rows are the input to the two money numbers on the platform: outcome billing (what a passed resolution rates at) and Agent ROI attribution (what a passed resolution is worth to you). This guide walks the full loop: author a rubric, stamp outcomes, read the results, and revise weekly.

Concepts: outcome rubric vs QA form

Orbit ships two grading surfaces, and they write different rows with different consumers:

	Outcome rubric	QA evaluation form
Attaches to	An AI agent	Human-agent conversations
Graded by	The LLM judge, automatically on every closed conversation	A reviewer, the AI auto-scorer, or the agent self-evaluating
Result	`passed` / failed, plus confidence and reasoning	Weighted 0–100 score per criterion
Consumed by	Outcome billing, Agent ROI, the Outcomes tab	QA leaderboard, coaching, the appeal lifecycle
Authored in	Agents → [agent] → Outcomes, or `POST /agents/{agentId}/outcomes/rubrics`	Quality → Evaluations → Scorecards

The distinction that matters: a conversation only becomes a billable or ROI-attributed outcome when an outcome-rubric row says passed = true. QA forms grade how a human handled a conversation; they never feed billing or ROI. See QA Evaluations for that pipeline, and Outcome-based AI-agent billing for what a passed row does once outcome charging is enabled.

Author a rubric

In the dashboard

Open Agents, choose an agent, and open the Outcomes tab.
Choose Add rubric.
Give it a name that states the outcome, for example “Resolved user issue” or “Appointment booked”.
Write the success criteria in plain-language markdown. The judge reads this text verbatim, so describe exactly what a passing conversation looks like: what the agent must establish, what counts as resolution, and what must not happen.
Save. Every conversation that closes from that point on is scored against the rubric automatically.

The tab lists each rubric with a colour-coded pass-rate pill and a daily pass-rate trend chart (7/30/90-day windows), and shows an outcome P&L panel once outcomes exist. An agent holds at most 20 rubrics; the Add rubric button disables at the limit. Criteria that score well are specific and observable. Compare:

Weak: “The agent was helpful.”
Strong: “The customer’s issue is named, a resolution or a clear next step is given, and the customer confirms or stops responding after the resolution. A conversation that ends in an escalation to a human does not pass.”

Over the API

cURL

curl -X POST "https://api.orbit.devotel.io/api/v1/agents/agent_2k9xd4/outcomes/rubrics" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Appointment booked",
    "criteria_md": "The conversation ends with a confirmed date and time, the customer explicitly agrees to it, and no open question remains."
  }'

Field	Rules
`name`	Required, 1–200 characters.
`criteria_md`	Required, 1–10,000 characters of markdown. The judge reads it verbatim.
`evaluator_prompt`	Optional, max 8,000 characters. Omit to use the built-in judge prompt.
`scope`	Optional, defaults to `agent`. Org-wide scopes (`inbox`, `voice`, `org`) use the org-scoped rubric routes.

The 201 response returns the stored rubric: id, agent_id, name, criteria_md, evaluator_prompt, and timestamps. The id is the rubric’s stable key — outcome rows, per-rubric rates, and ROI overrides all reference the id, so renaming a rubric never re-prices its history or breaks its stats series. List rubrics with GET /agents/{agentId}/outcomes/rubrics (oldest first), update with PATCH .../rubrics/{rubricId} (omitted fields are preserved), and delete with DELETE, which stops new evaluations against the rubric. Writes require the owner, admin, or developer role and are audit-logged. A 21st rubric on one agent returns 422, and an unknown agent id returns 404.

Per-rubric rates

Rates are integer micro-cents — one cent is 1,000,000 micro-cents — configured on the organization, not on the rubric:

defaultRateMicroCents applies to any passed outcome without an override (default 500,000, i.e. $0.005).
perOutcomeRateMicroCents keys a rate to a rubric id, e.g. { "rub_8f3k2m9p1q": 2000000 } prices that rubric’s passed outcomes at $0.02.

Both sit on GET / PUT /api/v1/billing/outcome-pricing, require the owner, admin, or billing role, and are bounded at $10.00 (1,000,000,000 micro-cents) per outcome. Nothing charges until the workspace’s outcome-charging opt-in on the same resource is enabled; Outcome-based AI-agent billing covers the metering side in full. Note the two separate knobs: the billing rate above is what a passed outcome rates at when outcome charging is on, while the ROI config (below) is what a passed outcome is worth to you in your margin math.

Stamping outcomes

Automatic scoring by the judge

When a conversation closes, the judge evaluates it against each of the agent’s rubrics and writes one row per rubric with passed, confidence (0–1), and reasoning. Read a single conversation’s rows:

cURL

curl "https://api.orbit.devotel.io/api/v1/agents/conversations/conv_9d2mqa/outcomes" \
  -H "X-API-Key: $ORBIT_API_KEY"

Each row carries the rubric name and scope, the passed flag, the confidence, the reasoning text, and who evaluated it. Confidence is advisory: it tells you which rows to spot-check in your weekly loop, and no billing or ROI number gates on it.

Imported conversions

When the business result happens outside the conversation — the SMS deep-link led to a purchase, the reminder produced a show-up — stamp the outcome on the message that drove it:

cURL

curl -X POST "https://api.orbit.devotel.io/api/v1/messages/msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d/feedback" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "outcome": "confirmed" }'

outcome is confirmed or unconfirmed; the Twilio wire field Outcome is accepted as an alias. The endpoint is idempotent: resending the same outcome is a no-op, and flipping the outcome refreshes the feedback timestamp for your audit trail. An unknown message id returns 404. For imports from CRMs, mobile-measurement partners, or attribution vendors, the bulk endpoint takes 1 to 1,000 rows per call and persists them in a single update:

cURL

curl -X POST "https://api.orbit.devotel.io/api/v1/messages/feedback/bulk" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      { "id": "msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d", "outcome": "confirmed" },
      { "id": "msg_7b3d91e05c2a4f68a1d0c2e4b6a8f012", "outcome": "unconfirmed" }
    ]
  }'

The Twilio-shaped envelope is accepted as an alias for the same import:

{
  "Items": [
    { "Sid": "msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d", "Outcome": "confirmed" }
  ]
}

The response carries updated[] (the ids the import actually touched), not_found[] (ids you supplied that matched no message in this workspace), and a summary counts block. A non-empty not_found still returns 200 — the batch as a whole succeeded, and reconciling the leftovers is yours: they typically come from another workspace, redacted rows, or messages older than your retention window.

Reading results

Pass-rate stats

GET /api/v1/agents/{agentId}/outcomes/stats?days=30 returns one bucket per rubric per day: total evaluations, passed_count, and pass_rate percent, joined with the rubric name. The Outcomes tab renders these as the pass-rate pill and the trend chart. days accepts 1–90 and defaults to 30.

Outcome P&L

GET /api/v1/agents/{agentId}/outcomes/pnl?days=30 rates the agent’s passed outcomes through your configured micro-cent rates and returns the cost-vs-value picture:

{
  "enabled": true,
  "currency": "USD",
  "totalOutcomes": 61,
  "totalMicroCents": 1525000000,
  "lineItems": [
    {
      "rubricId": "rub_8f3k2m9p1q",
      "rubricName": "Appointment booked",
      "count": 61,
      "rateMicroCents": 2000000,
      "subtotalMicroCents": 122000000
    }
  ],
  "days": 30
}

enabled mirrors the workspace’s outcome-charging opt-in: when it is false the same figures render as Projected rather than Billed in the dashboard panel. The endpoint rates already-recorded outcomes and never moves money — calling it bills nothing.

Feeding Agent ROI

Agent ROI answers “is this agent returning more than it costs?” by pricing passed outcomes against the agent’s model-call spend. Its value side is your assumption, configured on PUT /api/v1/insights/agent-roi/config:

default_outcome_value_cents — whole cents per passed outcome with no override; null prices unpriced outcomes at $0.
per_rubric_value_cents — per-rubric overrides in cents, keyed by the same rubric id.

Because both configs key on the rubric id, the rate you set for billing and the value you set for ROI stay attached to a rubric across renames. Once a rubric is passing conversations, set its value-per-outcome there and the ROI summary, daily trend, and per-agent margin pick the rows up without further configuration.

The weekly loop

Read the trend. Open the Outcomes tab or call outcomes/stats, and look for rubrics whose pass rate moved since last week.
Spot-check the borderline. For rows you doubt, open the conversation’s outcome rows and read the judge’s reasoning against the transcript; low confidence rows are the ones to pull first. For human-graded review of conversations, the QA Evaluations pipeline is the right tool.
Revise the criteria, keep the identity. Tune the rubric with PATCH .../rubrics/{rubricId} and a revised criteria_md. Future closed conversations are judged against the new text while historical rows stand as judged at their time, and because the rubric id is unchanged its pass-rate series, per-rubric rate, and ROI override all keep applying. Avoid deleting and re-creating a rubric to “version” it: the replacement gets a fresh id and starts with empty history and no rates.
Close the loop. Re-read the P&L and the ROI summary over the same window and compare against last week before touching criteria again.

Limits and roles

Rubric writes (POST / PATCH / DELETE on /outcomes/rubrics): owner, admin, or developer; audit-logged. Rubric, stats, and P&L reads: agents:read.
Outcome pricing config (GET/PUT /api/v1/billing/outcome-pricing): owner, admin, or billing.
Agent ROI config and dashboard: owner, admin, developer, or billing.
At most 20 rubrics per agent (422 over the limit); GET rubrics lists up to 100, oldest first.
days on stats and P&L: 1–90, default 30.
Rates are integer micro-cents bounded at $10.00 per outcome; non-integer or over-cap values are rejected with 422.
Confidence is advisory, and every read in this guide is money-free: nothing here bills, charges, or re-scores a conversation.

API Reference

All Endpoints

Agent custom outcome metrics: author rubrics, stamp outcomes, read ROI

Agent custom outcome metrics

Concepts: outcome rubric vs QA form

Author a rubric

In the dashboard

Over the API

Per-rubric rates

Stamping outcomes

Automatic scoring by the judge

Imported conversions

Reading results

Pass-rate stats

Outcome P&L

Feeding Agent ROI

The weekly loop

Limits and roles

See also

​Agent custom outcome metrics

​Concepts: outcome rubric vs QA form

​Author a rubric

​In the dashboard

​Over the API

​Per-rubric rates

​Stamping outcomes

​Automatic scoring by the judge

​Imported conversions

​Reading results

​Pass-rate stats

​Outcome P&L

​Feeding Agent ROI

​The weekly loop

​Limits and roles

​See also

Agent custom outcome metrics

Concepts: outcome rubric vs QA form

Author a rubric

In the dashboard

Over the API

Per-rubric rates

Stamping outcomes

Automatic scoring by the judge

Imported conversions

Reading results

Pass-rate stats

Outcome P&L

Feeding Agent ROI

The weekly loop

Limits and roles

See also