Skip to main content

​
Read the Insights dashboards as one decision picture

The Insights section of the Orbit console hosts about thirty dashboards — agent comparison, agent ROI, account scores, anomalies, attribution, contact reasons, containment, conversations, costs, funnels, goals, high-friction sessions, journey paths, language quality, LLM spend, media mix model, and more. Each one has its own guide. This one is different: it walks the cross-dashboard loop an operator actually runs — compare two AI agents on ROI, read the funnel per campaign, check cohort retention for the winning variant, and set a KPI alert so the answer stays watched. The point is that these pages are not thirty reports; they are one operating picture viewed from different angles. Comparison and ROI tell you which variant won. Cohorts tell you whether the win held after entry. Funnels and attribution tell you where the conversion credit belongs. Alerts keep the number on a tripwire after you close the tab. Read in that order, the loop turns a pile of dashboards into a decision with a monitoring rule behind it. This guide is written for an exec or operations lead who owns the outcome rather than the instrumentation. Each section names the page, the question it answers, and the API surface behind it. The per-surface guides — Compare AI agents with percentile benchmarks, Cohort retention, Funnels & Retention, Multi-touch attribution, KPI alerts — carry the field-by-field detail; read this one for the order of operations.

​
1. Model semantics — what predictive event each page reads

Before the loop, map the vocabulary. The Insights pages read two families of data — observed aggregates (containment, cost, revenue, funnel counts, retention rates computed from your traffic) and predictive model scores (probabilities assigned per contact and rolled up to accounts). Confusing the two produces bad decisions: a containment rate is measured; a churn-propensity score is predicted. The first is fact over a window, the second is an inference over a model. The predictive side runs four model families, each answering a different question about a contact: Every score is also drift-checked: GET /api/v1/cdp/predictive-models/{model}/drift compares the live scoring population against the model’s training distribution (Population Stability Index), so a page that reads a model score is reading a number whose input health you can inspect. Full request and response shapes are in the CDP analytics API reference; the operator-level reading of the account rollup is in Account scores. The observed side needs no model: funnels count subjects through ordered CDP events, cohort retention counts first-fires and returns, and the agent-comparison/ROI surfaces aggregate cost, containment, and attributed revenue from conversation traffic. The loop below mixes both deliberately — comparison (observed) picks a winner, cohorts (observed) confirm durability, funnels and attribution (observed) assign credit, and the predictive scores tell you which way the population underneath is drifting while you act.

​
2. The agent-comparison walkthrough — pick two agents by version, compare containment, quality, and ROI

The loop usually starts with a variant question: which of these two agent versions should keep the traffic? Open Insights → Agent comparison.
  1. Pick the window first — a 30 d window for a deployment decision, 7 d if you are reacting to a change this week. Percentiles are recomputed per window, so reading before fixing the window reads a different population.
  2. In the agent picker, select the two versions you are deciding between (by version tag or name — the picker filters by name/id and shows each agent’s conversation count). The comparison table needs at least two selections.
  3. Read these columns per cohort of traffic — containment (share resolved without a human), resolution, escalation, then the money columns the Agent ROI attribution pipeline feeds: cost, attributed revenue, margin, and margin %.
  4. Check the band badge before the raw number. Each metric cell carries a percentile rank and a quartile band against the org population, direction-aware — “top quartile” on escalation means low escalation. Then confirm with the conversation count next to the agent name: a top-quartile badge on four conversations is provisional.
  5. Sort by margin to find which version is actually paying; sort by cost to see which version is consuming budget per resolution. If containment and resolution both hold while cost drops, the cheaper version is the winner — if not, decide how much margin a point of resolution is worth.
For a report outside the console, snapshot the same comparison over the API:
The response carries the metric catalogue (key, label, unit, higherIsBetter), the population flags (population_size, population_truncated), and per-agent benchmark fields (value, percentile, rank, sample_size, band) — exact shapes in the Agent comparison & benchmarks section of the Insights API reference. Exit criterion for this step: you can say, in one sentence, which version wins on which metric at what volume. If you cannot, the volume is too thin or the window is wrong — fix that before reading cohorts.

​
3. Cohort reading on a per-campaign basis — which send cohort drove which retention delta

A comparison win is only a snapshot at the aggregate level; the question that decides a rollout is whether the cohorts the variant touched stayed better after entry. That is a cohort read, run per campaign rather than as one blended account curve. Build the read per campaign:
  1. Open Insights → Cohort Retention (or the retention grid on Insights → Funnels & Retention).
  2. Set the cohort event to the first-fire marker the campaign produced for the variant’s cohort — for a send-driven flow this is the campaign’s entry event (e.g. may_campaign_enrolled), so each row of the grid is “contacts who entered via that campaign that week.” A cohort only means something if the cohort event is a non-repeating milestone; if your tracking plan fires the entry event on every touch, the subject joins the earliest-fire bucket and the per-campaign read dissolves — keep the milestone discipline from the cohort guide.
  3. Set the return event to the habit or value predicate you actually retain on (Logged In, Order Completed, Invoice Paid).
  4. Run weekly buckets over 12 periods. Read the grid two ways: down the W1 column asks whether later campaign cohorts retain better than earlier ones (a variant that moved retention shows up as a step-change in that column at the cohort week the variant started taking traffic); across a row asks how that campaign’s cohort decays — a variant that front-loads value should show a higher W1/W2 floor even if the tail converges.
The per-campaign trick is holding the return predicate fixed while swapping the cohort event across campaigns. Change the return event and the delta you attribute to the campaign changes definition mid-read; change the cohort event and the same return predicate now answers “did this campaign’s entrants come back?” on like-for-like terms. Over the API this is one body per campaign:
Each cohort row carries cohort_period, cohort_size, and retention_rate/retained vectors where index 0 is the entry bucket (always 100%). Right-censoring is explicit: cohorts too young to have reached a period simply stop there rather than printing a false-zero dip — read the censor note under the grid before treating a tail as a signal.

​
4. Funnel reading tied to the attribution read below it

Now the flow question: the winning variant moved cohort retention — where in the flow did it move it, and which campaign gets the credit? These are two endpoints over the same CDP event stream, and reading them together is the whole point:
  • The funnel (POST /api/v1/cdp/analytics/funnel) measures your definition: the ordered steps you name, entry to close. Run it twice — once on the full population, once on the campaign cohort — and the step where the campaign cohort’s step-conversion outruns the population’s is where the campaign earned its delta.
  • Multi-touch attribution (POST /api/v1/cdp/analytics/attribution and .../revenue-attribution) measures credit: of the conversions that happened, how does the chosen model (first-touch, last-touch, linear, time-decay, U-shaped) split them across the channels and campaigns that touched the converters. This is the read below the funnel — the funnel says where subjects stopped; attribution says who gets the credit for the ones that didn’t.
Tie them in practice:
Read the funnel’s step conversion column next to the attribution rollup per campaign: if the campaign cohort converts the entry step at 58% where the population converts it at 42%, and the attribution model credits that campaign 61% of closed conversions under time-decay, you have one story told twice — the cohort is entering at a higher rate and the campaign is earning the credit for the close. If they disagree (funnel says the mid-step broke, attribution still credits the campaign), the credit belongs to touchpoints around the break, which usually means the campaign introduces well but the flow leaks after it — that sentence is the difference between “scale the campaign” and “fix step two, then scale.” Model choice is the attribution guide’s territory — Multi-touch attribution carries the decision flow between first-touch, last-touch, linear, time-decay, and U-shaped; the loop only requires you to run the same window under the same model on both sides of a comparison, because every model normalizes to 1.0 credit per conversion and re-weights rather than re-samples.

​
5. KPI alerts — pin the result on a tripwire

The loop ends where it becomes continuous: the number you just decided on should fire a notification the week it moves. Insights → Alerts (/insights/alerts) defines org-scoped rules over the cross-pillar KPIs — CSAT, NPS, CES, LLM spend, and AI containment — each either a fixed threshold (lt/lte/gt/gte over a trailing window) or an anomaly detector against the metric’s own 28-day baseline. Create the rule that matches the decision you made:
  • Rolled out the cheaper agent version → guard AI containment against a drop (the margin win is not worth a containment regression).
  • Scaled the campaign on a funnel read → guard CSAT or NPS against a drop if the higher-volume cohort starts degrading experience.
  • Moved model or spend → guard LLM spend with a threshold cap.
Dashboard path: Insights → Alerts → Create rule, pick the metric, pick threshold or anomaly, set comparator/threshold/window for threshold mode, enable. The API carries the same fields:
A fired rule pushes a dashboard notification with a deep link to the metric page (/insights/containment, /insights/surveys, /insights/llm-spend); the event feed (GET /api/v1/analytics/kpi-alert-rules/events, last 50) gives you the same fire programmatically. The alert evaluates against exactly the aggregates the dashboards show — so the number you read in step 2 is the number the rule guards. Rules are read-only over your data: they notify; they never send traffic. Full field, lifecycle, and cooldown semantics are in KPI alerts; the transport-side counterpart (delivery rate, volume, spend) is in Usage & delivery anomaly alert rules.

​
6. Worked scenario — “did the May campaign move my weekly active cohort?”

Put the whole loop on one concrete question. Scenario: the May campaign ran a new onboarding flow through agent version ag_onboard_v4; you need to know whether the campaign actually moved weekly active retention, and you want the answer monitored. Step 1 — compare the variant. Open Insights → Agent comparison, 30 d window, select ag_onboard_v3 and ag_onboard_v4. Suppose v4 holds resolution in the same quartile while cost drops a band — a margin win; suppose containment also holds. The comparison’s verdict: roll v4 to the May campaign traffic. If containment had dropped a band, the rest of the loop would not run — decision stops here. Step 2 — read the campaign cohort. On Insights → Cohort Retention, cohort event may_campaign_enrolled, return event Logged In, weekly, 12 periods. The read you need is down the W1 column: if cohorts entering from the May campaign hold W1 at, say, 34% where the pre-campaign cohorts held 28%, the campaign moved weekly-active retention by ~6 points and older cohorts stay healthy — the right-censoring note tells you the youngest cohorts are honestly missing their tail, not dipping. Step 3 — locate the delta in the flow; assign the credit. Run the funnel Campaign Sent → Activated → Purchased over the same window; the step-conversion column says the May cohort’s entry step converted better. Run POST /api/v1/cdp/analytics/attribution with conversion_event: "Purchased", model: "linear", same lookback — the May campaign’s credit share in the rollup is the number you quote when you say “the May campaign drove it.” Funnel says where; attribution says whose. Step 4 — export the cohort for the stakeholder email. The per-campaign curl from section 3 gives you the grid; the digest a stakeholder reads is cohort week, size, W1, and latest observable period — extract it with the same jq line and paste it into the report. Step 5 — write the alert threshold. The decision “W1 moved to ~34%” becomes a tripwire: a containment floor for the new agent version, and a CSAT floor for the now-larger weekly active cohort. Create both under Insights → Alerts, or over the API:
Now the loop closes: the campaign is scaled, the cohort read is in the weekly digest, and the first regression in containment or CSAT fires a notification with a deep link before the next read. That is the difference between reading the Insights dashboards and operating them.

​
Troubleshooting the loop

​
See also