Skip to main content

​
Tenant migrations: drift, re-runs, and failures

Orbit gives each organization its own database schema, and schema evolution applies against the tenant ledger row by row — not against a shared platform schema. The Tenant migrations console is where you see whether every tenant in your fleet carries the current required set, and re-run the missing ones safely. This guide is the walkthrough for that console: what drift actually measures, why the scan window is bounded, when to re-run one tenant versus the whole fleet, how failures drain through the failed-runs list, and how every manual re-run lands in the audit log. This page is tenant-schema migrations only — it has nothing to do with the data-import migration jobs (importing contacts from an external system, covered in run a migration from Settings → Migrations), nothing to do with dbt/ELT-style analytics migrations, and nothing to do with migrating data between Orbit services. It answers one question: “is every tenant schema on the required schema list, and if not, run it.”

​
1. Where the page lives

Open the admin console and go to Migrations in the sidebar. The page is visible to anyone with admin-console access, and write actions (re-run single, re-run bulk) are guarded behind the same server check everywhere else. The console is split into two sections stacked on each other:
  • Tenants — every row is one tenant organization, with its applied count, expected count, drift count, and latest-applied timestamp. This is the workhorse: whenever drift exists, it appears here.
  • Failed runs — the mitigation FIFO, explained below (§4). If the page can render it, every migration attempt the runner logged is listed, newest first.
Both sections refresh on a 60-second interval; the buttons below settle their own mutation and then invalidate the cached queries, so the page is eventual-consistent. If you re-run and expect to see the drift clear within minutes, refresh it before you declare it solved.

​
2. What the counters mean (and the scanned window)

Four stat cards at the top of the page plus per-row fields tell a different aspect of the same ledger:
  • Expected migrations — the total count of schema-migration entries every tenant should have applied, as computed by the platform’s required list. When this number rises (because the platform added a new tenant-schema migration), drift is bred everywhere, everywhere at once.
  • Tenants scanned — the number of tenant organizations the read query probed before rendering the page. This is a bounded window, not the full platform: the page probes the newest first 500 organizations and reports drift only within that window, because probing every organization on every page load scales linearly with the platform’s tenant count, which grows. When the platform is smaller than the window cap, the counts equal; once it grows past the cap, the drift badge tells you only about the window, not the whole platform — a drifting tenant outside the scanned window never surfaces here. The page discloses the relationship explicitly when it matters: a warning line above the cards reads “Showing the N most-recently-created tenants of M total,” so a drift count of zero means the window converged, not that the platform converged.
  • Tenants drifting (in window) — the sum over the scanned rows of drift > 0. This is the actionable count, and it is computed by reading drift on each row, not by computing drift server-side — same number, consistently measured.
  • Public schema drift — the platform’s global scope runs its own ledger, independent of the tenants. If it drifts, that divergence is reported the same way a tenant drift would be, and the repair passes through the same console.
Per-row, you read four fields:
  • Applied — number of required migrations the tenant carries (for example 12 / 14). A parenthetical (+n superseded) after the count flags ids the platform no longer lists — renamed or renumbered entries the tenant already applied, and therefore not drift: the required set and the applied set drift over each other only on required ids, not on histories.
  • Drift — how many of the platform’s required ids are missing from the tenant. A success badge (green) means zero; a warning badge (amber) means the row is low drift for a single operator push; an error badge means the missing set is deep enough that an operator triage script would have already flagged it. Hovering the drift badge enumerates the first few missing migration ids; the full set is available in the audit-log details for any run you trigger.
  • Latest applied — the timestamp of the tenant’s most recent applied entry, or ”—” when the tenant has never carried one.
Every row also carries a tenant-id copy button, because the schema name (the tenant_<id> schema) never shows the full id inline.

​
3. Re-run: single row vs the whole drifting window

Two actions exist, and both go through the same guarded path:
  • Re-run on a single row — enabled only on rows with drift > 0 (rows at drift 0 render a static “Up to date” badge, so the button can’t fire a no-op run against an already-converged tenant and pollute the audit ledger). Clicking it opens a confirm dialog that names the exact tenant and schema it will run against, then calls the same locked helper the tenant bootstrap path uses. Concurrent clicks by two operators converge anyway: the helper picks up a database advisory lock, and every migration entry gets applied at most once.
  • Re-run all drifting — enabled only while at least one scanned row drifts. This bulk action takes the exact tenant ids currently showing drift > 0 inside the scanned window (nothing outside the window — and that scoping matters, because the bulk action without the window would hit drifting tenants that the page couldn’t display). Each id then runs through the same locked single-row path, so the batch is safe even if a tenant converges mid-batch or a second operator re-runs it before the batch lands.
Both mutations emit a success toast with the succeeded count, or a failure toast naming how many failed, and then the page re-fetches the drift counters. On a healthy platform, a bulk re-run converges the entire scanned window within a few polish passes. What drift actually means for the re-run: drift is the count of required ids absent from the tenant’s applied set — re-running is therefore the application of exactly that set, and nothing else. It never re-applies ids the tenant already carries, and it never touches superseded ids. Rerunning with drift at zero is harmless but pointless; the console disables it by design.

​
4. Resolve failed migration runs

The Failed runs section at the bottom lists every row in the platform’s failed-migration ledger, ordered by the newest attempt. Each row shows:
  • Migration id — the numbered entry that failed.
  • Scope — either public (platform-wide) or the tenant_<id> schema it failed against.
  • Error — the last recorded failure message, truncated to the first line.
  • Attempts — how many times the runner has logged this failure.
  • Last attempted — when it was last touched.
The Retry all button (enabled whenever at least one row exists) re-dispatches each recorded failure through the same guard: platform-wide rows go through the platform’s public runner, and tenant-scoped rows go through the same single-tenant runner the Re-run button uses. Rows that succeed on retry are cleared from the list; rows that fail again remain with their attempts incremented. If the same failure recurs across retries, the error column is the evidence you escalate with — see §5. Nothing on this page is lost by retrying: the list is a FIFO of recorded failures with no independent persistence of its own — a row disappears exactly when the migration it names lands.

​
5. The audit trail: every run is ledgered

The console never assumes success — every mutation on it writes to the organization’s audit log with a dedicated action, and the Audit log page (/guides/audit-log) is where you read them back. Three distinct actions make up the re-run family:
  • admin.migrations.run_single — fired by the single-row Re-run button.
  • admin.migrations.run_bulk — fired by the Re-run all drifting button, one entry per batch.
  • admin.migrations.retry_failed — fired by the Retry all button over the failed-runs list.
Each row captures the tenant id (or ids) it ran against, the human operator (or API key, if you scripted it), and the structured detail — invoked ids, the succeeded/failed split, and any failure detail the runner logged. Because the log is append-only, you can always re-derive “who re-ran what when” from Settings → Audit log without guessing, and the same actions are available to a SIEM pipeline subscribed to the audit.log.created webhook event. When you escalate a stuck failure — a tenant whose drift doesn’t converge, or a failed-run row that keeps re-failing the same migration id — capture three things from the console and the log: the tenant id (the copy button on the row puts it into your clipboard), the audit-log row for the run (expand it for the attempted/succeeded/failed split, and the exact migration ids it hit), and the failed-run row’s error + attempt count. With those, support can answer “why did this id fail” in one lookup without reconstructing the sequence from your account history.

​
How to run this safely

  1. Open the page. Check the scan-window warning above the cards; if the platform is bigger than the scan window, treat “zero drift” as “the window converged.”
  2. Re-run single or re-run bulk. Prefer bulk when more than a handful drift; prefer single when you can name the one outlier.
  3. Check the failed-runs list. When a run fails, the row lands here with the error — retry the recorded failures, and read the error text before deciding to proceed.
  4. Audit after the run. Open Settings → Audit log and filter to the admin.migrations.run_* family; confirm the expected tenants appear in the succeeded set.
  5. Escalate when the same id keeps failing. The audit log’s structured detail is the evidence a support ticket needs; a narrated story without a row id never converges.

​
See also