Skip to main content

​
The QM coaching loop: queue, notes, scorecards, effectiveness

Quality scoring tells you which conversations went badly. Coaching is what you do about it. Devotel Orbit ships the whole loop in one surface: a supervisor queue of the conversations the LLM judge flagged, coaching notes attached to a specific conversation with rubric references, agent acknowledgement so the feedback is confirmed read, and a measured before/after report that tells you whether the coaching actually moved the agent’s numbers. This page covers the asynchronous coaching loop on digital conversations (chat, email, WhatsApp, social). It is a different surface from the live whisper coaching in the softphone, which feeds suggestions to a voice agent while a call is still in progress. Base path: /api/v1/inbox Dashboard surface: open the Inbox → Coaching page; agents read their own numbers on /me/scorecard.

​
The coaching loop

The loop runs on the LLM-judged rubric outcomes that power the quality-management program. Each step maps to an endpoint:
  1. Supervisor triages the queue. Scored conversations whose average rubric score is below your threshold (and with at least one failed rubric) surface in the coaching queue, lowest score first.
  2. Supervisor writes a coaching note. The note is attached to a specific conversation, references the failed rubrics, and may carry a suggested action.
  3. Agent acknowledges the note. The agent the note is addressed to marks it as read. Until acknowledged, the note sits open on the conversation.
  4. Scorecard tracks the agent. Per-rubric pass rates, a daily trend, and the agent’s top failure modes over a window (default 30 days).
  5. Effectiveness is measured. The report compares the agent’s response time, resolution rate, and CSAT in equal-length windows before and after the coaching event, and classifies each KPI as improved, regressed, or flat.
Response time is “lower is better”; resolution rate and CSAT are “higher is better.” The report bakes polarity in so a falling response time reads as an improvement, not a regression.

​
Triage the coaching queue

GET /inbox/coaching/queue returns conversations that scored below your threshold, ordered by lowest average score — so the worst come up first.
Query parameters: Each queue entry carries the conversation id, the agent it is assigned to, the average and minimum rubric scores, the failed/total rubric counts, and when it was last scored. A tenant with no QM scoring provisioned yet gets an empty queue with a 200 — not an error.

​
Attach and list coaching notes

A supervisor attaches a note to a specific conversation with POST /inbox/conversations/:id/coaching-notes:
The body’s agent_id must match the agent the conversation is assigned to — a mismatch returns 422 AGENT_MISMATCH, which keeps a note from being filed against the wrong teammate. Up to 10 rubric references attach to a note; the note text is capped at 5,000 characters. List a conversation’s coaching history — newest first — with GET /inbox/conversations/:id/coaching-notes:
Owners, admins, and supervisors see every note on the conversation; an agent sees only the notes addressed to them.

​
Agent acknowledges the note

The agent the note is addressed to marks it read with PUT /inbox/conversations/:id/coaching-notes/:noteId/acknowledge:
Only the addressed agent can acknowledge; any other caller gets 403 FORBIDDEN. The response stamps acknowledged_at and acknowledged_by_agent: true, which supervisors read back from the note list to confirm the feedback landed.

​
Read the per-agent scorecard

GET /inbox/agents/:agentId/scorecard (with ?days=30) returns the agent’s quality scorecard: overall totals, per-rubric pass rates sorted by lowest first, a daily score trend, and the top failure modes.
The top_failure_modes block lists the rubrics with a pass rate under 80% — the top of that list is the coaching focus. An agent may read only their own scorecard; owners, admins, and supervisors read any. The agent-facing view is documented in Read your own QM scorecard.

​
Measure coaching effectiveness

GET /inbox/agents/:agentId/coaching-effectiveness is the “did the coaching work” report. It compares three digital KPIs — response time, resolution rate, and CSAT — in equal-length windows before and after a coaching event:
  • Anchor. Pass anchor_at (ISO 8601) explicitly, or omit it to anchor on the agent’s most recent coaching note. If the agent has no notes and no anchor is supplied, the route 404s — there is nothing to measure against.
  • Windows. Symmetric, window_days long on each side (default 14). The after window is capped at “now” so a recent note never measures against a future, empty span; the response sets in_progress: true when that happens.
  • Per KPI. Each entry carries before, after, delta, pct_change, and a direction of improved / regressed / flat / insufficient_data. Response time is lower-is-better; resolution rate and CSAT are higher-is-better. A metric either window lacked data for reads as insufficient_data instead of a misleading zero-delta.
The report only counts the agent’s digital-channel conversations — voice coaching effectiveness on calls is reported separately by the voice coaching-plans surface.

​
The leaderboard (optional)

GET /inbox/leaderboard ranks every scored agent by rubric pass rate over the window:
The leaderboard is opt-in: an admin enables it under Settings → Quality Management. Until enabled, the route returns 403. Privacy is layered per caller:
  • Owners, admins, and supervisors always see real names.
  • Other callers see names only when the leaderboard_show_names org setting is enabled; otherwise every entry is pseudonymized to a stable Agent-a1b2c3d4 handle, so trend comparisons still work.
  • The caller’s own entry is always revealed with their real name.

​
Role matrix

Every capability in this page is a tenant-owned control: enable the leaderboard, set your queue threshold, and scope who sees names from your own organization settings. Nothing in the coaching loop sends anything to a customer — notes and scorecards live inside your workspace.