> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orbit.devotel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent custom outcome metrics: author rubrics, stamp outcomes, read ROI

> Define what a resolved conversation looks like for each AI agent, score outcomes automatically or import your own conversion data, and read pass-rate, P&L, and ROI from the same rows.

# Agent custom outcome metrics

Every AI agent in Orbit has an **Outcomes** tab where you define the business results you care about — resolved ticket, booked appointment, correct escalation — as **rubrics**. An LLM judge reads each closed conversation against every rubric and records a pass/fail row with a confidence score and a line of reasoning. Those rows are the input to the two money numbers on the platform: outcome billing (what a passed resolution rates at) and [Agent ROI attribution](https://docs.orbit.devotel.io/guides/agent-roi-attribution.md) (what a passed resolution is worth to you). This guide walks the full loop: author a rubric, stamp outcomes, read the results, and revise weekly.

## Concepts: outcome rubric vs QA form

Orbit ships two grading surfaces, and they write different rows with different consumers:

| | Outcome rubric | QA evaluation form |
| - | - | - |
| Attaches to | An AI agent | Human-agent conversations |
| Graded by | The LLM judge, automatically on every closed conversation | A reviewer, the AI auto-scorer, or the agent self-evaluating |
| Result | `passed` / failed, plus confidence and reasoning | Weighted 0–100 score per criterion |
| Consumed by | Outcome billing, Agent ROI, the Outcomes tab | QA leaderboard, coaching, the appeal lifecycle |
| Authored in | **Agents → \[agent] → Outcomes**, or `POST /agents/{agentId}/outcomes/rubrics` | **Quality → Evaluations → Scorecards** |

The distinction that matters: a conversation only becomes a billable or ROI-attributed outcome when an outcome-rubric row says `passed = true`. QA forms grade how a human handled a conversation; they never feed billing or ROI. See [QA Evaluations](https://docs.orbit.devotel.io/guides/quality-evaluations.md) for that pipeline, and [Outcome-based AI-agent billing](https://docs.orbit.devotel.io/concepts/outcome-pricing-model.md) for what a passed row does once outcome charging is enabled.

## Author a rubric

### In the dashboard

1. Open **Agents**, choose an agent, and open the **Outcomes** tab.
2. Choose **Add rubric**.
3. Give it a name that states the outcome, for example "Resolved user issue" or "Appointment booked".
4. Write the **success criteria** in plain-language markdown. The judge reads this text verbatim, so describe exactly what a passing conversation looks like: what the agent must establish, what counts as resolution, and what must not happen.
5. Save. Every conversation that closes from that point on is scored against the rubric automatically.

The tab lists each rubric with a colour-coded pass-rate pill and a daily pass-rate trend chart (7/30/90-day windows), and shows an outcome P\&L panel once outcomes exist. An agent holds at most 20 rubrics; the **Add rubric** button disables at the limit.

Criteria that score well are specific and observable. Compare:

* Weak: "The agent was helpful."
* Strong: "The customer's issue is named, a resolution or a clear next step is given, and the customer confirms or stops responding after the resolution. A conversation that ends in an escalation to a human does not pass."

### Over the API

```bash cURL theme={null}
curl -X POST "https://api.orbit.devotel.io/api/v1/agents/agent_2k9xd4/outcomes/rubrics" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Appointment booked",
    "criteria_md": "The conversation ends with a confirmed date and time, the customer explicitly agrees to it, and no open question remains."
  }'
```

| Field | Rules |
| - | - |
| `name` | Required, 1–200 characters. |
| `criteria_md` | Required, 1–10,000 characters of markdown. The judge reads it verbatim. |
| `evaluator_prompt` | Optional, max 8,000 characters. Omit to use the built-in judge prompt. |
| `scope` | Optional, defaults to `agent`. Org-wide scopes (`inbox`, `voice`, `org`) use the org-scoped rubric routes. |

The 201 response returns the stored rubric: `id`, `agent_id`, `name`, `criteria_md`, `evaluator_prompt`, and timestamps. The `id` is the rubric's stable key — outcome rows, per-rubric rates, and ROI overrides all reference the id, so renaming a rubric never re-prices its history or breaks its stats series. List rubrics with `GET /agents/{agentId}/outcomes/rubrics` (oldest first), update with `PATCH .../rubrics/{rubricId}` (omitted fields are preserved), and delete with `DELETE`, which stops new evaluations against the rubric.

Writes require the owner, admin, or developer role and are audit-logged. A 21st rubric on one agent returns 422, and an unknown agent id returns 404.

### Per-rubric rates

Rates are integer micro-cents — one cent is 1,000,000 micro-cents — configured on the organization, not on the rubric:

* `defaultRateMicroCents` applies to any passed outcome without an override (default 500,000, i.e. \$0.005).
* `perOutcomeRateMicroCents` keys a rate to a rubric id, e.g. `{ "rub_8f3k2m9p1q": 2000000 }` prices that rubric's passed outcomes at \$0.02.

Both sit on `GET` / `PUT /api/v1/billing/outcome-pricing`, require the owner, admin, or billing role, and are bounded at \$10.00 (1,000,000,000 micro-cents) per outcome. Nothing charges until the workspace's outcome-charging opt-in on the same resource is enabled; [Outcome-based AI-agent billing](https://docs.orbit.devotel.io/concepts/outcome-pricing-model.md) covers the metering side in full.

Note the two separate knobs: the billing rate above is what a passed outcome **rates at** when outcome charging is on, while the ROI config (below) is what a passed outcome is **worth to you** in your margin math.

## Stamping outcomes

### Automatic scoring by the judge

When a conversation closes, the judge evaluates it against each of the agent's rubrics and writes one row per rubric with `passed`, `confidence` (0–1), and `reasoning`. Read a single conversation's rows:

```bash cURL theme={null}
curl "https://api.orbit.devotel.io/api/v1/agents/conversations/conv_9d2mqa/outcomes" \
  -H "X-API-Key: $ORBIT_API_KEY"
```

Each row carries the rubric name and scope, the passed flag, the confidence, the reasoning text, and who evaluated it. Confidence is **advisory**: it tells you which rows to spot-check in your weekly loop, and no billing or ROI number gates on it.

### Imported conversions

When the business result happens outside the conversation — the SMS deep-link led to a purchase, the reminder produced a show-up — stamp the outcome on the message that drove it:

```bash cURL theme={null}
curl -X POST "https://api.orbit.devotel.io/api/v1/messages/msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d/feedback" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "outcome": "confirmed" }'
```

`outcome` is `confirmed` or `unconfirmed`; the Twilio wire field `Outcome` is accepted as an alias. The endpoint is idempotent: resending the same outcome is a no-op, and flipping the outcome refreshes the feedback timestamp for your audit trail. An unknown message id returns 404.

For imports from CRMs, mobile-measurement partners, or attribution vendors, the bulk endpoint takes 1 to 1,000 rows per call and persists them in a single update:

```bash cURL theme={null}
curl -X POST "https://api.orbit.devotel.io/api/v1/messages/feedback/bulk" \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      { "id": "msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d", "outcome": "confirmed" },
      { "id": "msg_7b3d91e05c2a4f68a1d0c2e4b6a8f012", "outcome": "unconfirmed" }
    ]
  }'
```

The Twilio-shaped envelope is accepted as an alias for the same import:

```json theme={null}
{
  "Items": [
    { "Sid": "msg_0f8e2a4c6d1b3e5a7c9d0e2f4a6b8c1d", "Outcome": "confirmed" }
  ]
}
```

The response carries `updated[]` (the ids the import actually touched), `not_found[]` (ids you supplied that matched no message in this workspace), and a `summary` counts block. A non-empty `not_found` still returns 200 — the batch as a whole succeeded, and reconciling the leftovers is yours: they typically come from another workspace, redacted rows, or messages older than your retention window.

## Reading results

### Pass-rate stats

`GET /api/v1/agents/{agentId}/outcomes/stats?days=30` returns one bucket per rubric per day: `total` evaluations, `passed_count`, and `pass_rate` percent, joined with the rubric name. The Outcomes tab renders these as the pass-rate pill and the trend chart. `days` accepts 1–90 and defaults to 30.

### Outcome P\&L

`GET /api/v1/agents/{agentId}/outcomes/pnl?days=30` rates the agent's passed outcomes through your configured micro-cent rates and returns the cost-vs-value picture:

```json theme={null}
{
  "enabled": true,
  "currency": "USD",
  "totalOutcomes": 61,
  "totalMicroCents": 1525000000,
  "lineItems": [
    {
      "rubricId": "rub_8f3k2m9p1q",
      "rubricName": "Appointment booked",
      "count": 61,
      "rateMicroCents": 2000000,
      "subtotalMicroCents": 122000000
    }
  ],
  "days": 30
}
```

`enabled` mirrors the workspace's outcome-charging opt-in: when it is false the same figures render as **Projected** rather than **Billed** in the dashboard panel. The endpoint rates already-recorded outcomes and never moves money — calling it bills nothing.

### Feeding Agent ROI

[Agent ROI](https://docs.orbit.devotel.io/guides/agent-roi-attribution.md) answers "is this agent returning more than it costs?" by pricing passed outcomes against the agent's model-call spend. Its value side is your assumption, configured on `PUT /api/v1/insights/agent-roi/config`:

* `default_outcome_value_cents` — whole cents per passed outcome with no override; `null` prices unpriced outcomes at \$0.
* `per_rubric_value_cents` — per-rubric overrides in cents, keyed by the same rubric id.

Because both configs key on the rubric id, the rate you set for billing and the value you set for ROI stay attached to a rubric across renames. Once a rubric is passing conversations, set its value-per-outcome there and the ROI summary, daily trend, and per-agent margin pick the rows up without further configuration.

## The weekly loop

1. **Read the trend.** Open the Outcomes tab or call `outcomes/stats`, and look for rubrics whose pass rate moved since last week.
2. **Spot-check the borderline.** For rows you doubt, open the conversation's outcome rows and read the judge's reasoning against the transcript; low `confidence` rows are the ones to pull first. For human-graded review of conversations, the [QA Evaluations](https://docs.orbit.devotel.io/guides/quality-evaluations.md) pipeline is the right tool.
3. **Revise the criteria, keep the identity.** Tune the rubric with `PATCH .../rubrics/{rubricId}` and a revised `criteria_md`. Future closed conversations are judged against the new text while historical rows stand as judged at their time, and because the rubric id is unchanged its pass-rate series, per-rubric rate, and ROI override all keep applying. Avoid deleting and re-creating a rubric to "version" it: the replacement gets a fresh id and starts with empty history and no rates.
4. **Close the loop.** Re-read the P\&L and the ROI summary over the same window and compare against last week before touching criteria again.

## Limits and roles

* Rubric writes (`POST` / `PATCH` / `DELETE` on `/outcomes/rubrics`): owner, admin, or developer; audit-logged. Rubric, stats, and P\&L reads: `agents:read`.
* Outcome pricing config (`GET`/`PUT /api/v1/billing/outcome-pricing`): owner, admin, or billing.
* Agent ROI config and dashboard: owner, admin, developer, or billing.
* At most 20 rubrics per agent (422 over the limit); `GET` rubrics lists up to 100, oldest first.
* `days` on stats and P\&L: 1–90, default 30.
* Rates are integer micro-cents bounded at \$10.00 per outcome; non-integer or over-cap values are rejected with 422.
* Confidence is advisory, and every read in this guide is money-free: nothing here bills, charges, or re-scores a conversation.

## See also

* [Outcome-based AI-agent billing](https://docs.orbit.devotel.io/concepts/outcome-pricing-model.md) — the charge gate and wallet-ledger side of passed rubric rows
* [Configure and read Agent ROI attribution](https://docs.orbit.devotel.io/guides/agent-roi-attribution.md) — pricing resolved outcomes into cost-vs-revenue margin
* [QA Evaluations](https://docs.orbit.devotel.io/guides/quality-evaluations.md) — the human-graded scorecard pipeline for coaching


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.