Troubleshooting: ENCRYPTION_FAILED (5xx) and DECRYPTION_FAILED (502)

Orbit’s encrypt-everything envelope seals tenant data at rest under AES-256-GCM with an enc:v1 tag. Two error codes surface that envelope:

Write side — ENCRYPTION_FAILED, a transient 5xx raised when the envelope could not seal the write (IV or authTag generation failed, or the crypto runtime was unavailable on the region serving your request).
Read side — DECRYPTION_FAILED, a 502 raised when the envelope could not open a stored ciphertext — the platform master key is missing, or a key rotation left a stale enc:v1 ciphertext whose authTag no longer matches.

{
  "error": { "code": "DECRYPTION_FAILED", "status": 502 },
  "meta": { "request_id": "req_5f2c9a" }
}

During a rotation window you may see the 502 until the envelope re-synchronizes; affected reads self-recover once the rotation completes, with no action from you. This page covers the platform master-key / rotation-stale-ciphertext class. The BYOK-enforced sibling — where your own customer-managed key’s lifecycle refuses the call with 409 BYOK_KEY_UNAVAILABLE — has its own cause table on BYOK key unavailable / decryption failed. Both codes are listed in the Error Code Reference.

BYOK is recorded posture, not the encryption key for tenant data: all tenant data at rest is sealed by Orbit’s platform-managed envelope regardless of your BYOK config’s state. So an ENCRYPTION_FAILED / DECRYPTION_FAILED from a tenant that runs without BYOK is a master-key class problem, and a tenant with BYOK sees the same two codes when BYOK is not enforced on the surface. Rotation of the BYOK reference never re-encrypts tenant data — registration and enforcement only record your compliance posture.

Symptom

A write returns a transient 5xx with code ENCRYPTION_FAILED — the envelope refused to seal the request payload (embeddings, webhook secrets, encrypted fields).
A read returns 502 with code DECRYPTION_FAILED — the stored envelope pointed at a stale or unavailable key, typically during or just after a key-rotation window.
The read error clears on its own once the crypto envelope re-synchronizes after rotation; the same 502 persisting past the rotation window means the ciphertext is genuinely stale.

Reproduce and capture the handle

Capture the request id. Read meta.request_id from the error envelope — it is the durable handle for escalation.
Identify the failing row. Pull the record the failure hit — the webhook delivery row (GET /api/v1/webhooks/{endpoint_id}/deliveries) or the verify-history entry (GET /api/v1/verify/history) that failed.
Re-request with the same request id: check whether the error is already gone (rotation completed) or persists.

curl -sS -X POST "https://orbit.devotel.io/api/v1/verify/history" \
  -H "Authorization: Bearer $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Request-Id: req_5f2c9a" \
  -d '{"limit": 1}'

Use the captured request_id as the X-Request-Id header so the platform can correlate the retry against the original failure. Verify your BYOK key posture in the dashboard at Settings → Security → BYOK (the Customer-Managed Keys section) — the state and enforced values there tell you whether the BYOK sibling runbook or this page applies. The lifecycle walkthrough is in the Customer-Managed Keys guide.

Cause table

Cause	How to confirm	Fix
Platform master key missing	Tenant runs without BYOK and the error persists with no rotation in flight — the platform’s master key was unreachable for that write/read.	Retry once with backoff; if it persists, escalate with `meta.request_id`. The master key is platform-side — nothing in your account changes it.
Rotation in flight, stale ciphertext	Reads return 502 during an announced rotation window; the stored `enc:v1` envelope’s authTag was sealed under the pre-rotation key. Reads self-recover as the re-encryption sweep lands.	Wait for the rotation window to close, then re-read once. If any affected read still 502s after the window, escalate — the stale ciphertext needs re-encrypting onto the current key, which only the platform can perform.
Crypto runtime unavailable on the region	`ENCRYPTION_FAILED` 5xx on writes only, transient; the libsodium / WebCrypto implementation was unreachable on the runtime region that served the request.	Transient-class: retry with backoff honoring `details.retry_after`. Escalate only if retries exhaust.

Recovery ladder

Work these in order:

Class the code against the deterministic / transient / conditional taxonomy. A 5xx ENCRYPTION_FAILED is transient — retry with exponential backoff, honoring details.retry_after when the error response carries it. A 502 DECRYPTION_FAILED is conditional: re-reading the same ciphertext during a rotation window fails until the envelope re-synchronizes, then succeeds.
Verify your BYOK key posture in the dashboard at Settings → Security → BYOK — confirm whether a lifecycle event (rotate, revoke, re-activate) lines up with the first failure. A rotated BYOK reference does not re-encrypt tenant data, but the timeline points you at the right runbook.
If the 502 persists past the rotation window, escalate at support@devotel.io with the meta.request_id — it is the durable handle. Stale ciphertext is re-encrypted onto the current key on the platform side; no retry or re-write from your side can do it.

What NOT to do

Do not re-write with a different payload. The envelope failure is not payload-bound — re-submitting a modified payload hits the same master-key condition and burns idempotency keys.
Do not bypass BYOK rotation. Rotating again to “force a re-sync” while a rotation is in flight compounds the failure — resolve the current rotation first. The downgraded STIR/SHAKEN attestation has its own path on STIR/SHAKEN attestation downgrade; do not treat an envelope failure as an attestation problem.
Do not re-paste the same ciphertext into a retry loop. On DECRYPTION_FAILED the stored envelope pointed at a pre-rotation key; re-reading it returns the same 502 every time until it is re-encrypted.
Do not register a BYOK key as a fix for ENCRYPTION_FAILED. BYOK is recorded posture, not the encryption key for tenant data — with or without BYOK the platform-managed envelope seals the write, so the refuse below is not resolved from your KMS.

When to escalate

Escalate to support@devotel.io when:

Any 502 DECRYPTION_FAILED persisting past the rotation window — the stale ciphertext needs re-encryption onto the current key.
ENCRYPTION_FAILED 5xx outlasts backoff retries — the crypto runtime or master-key availability on the region is platform-side.

Include the request_id from meta.request_id, the UTC timestamp, and the failing surface (write path or the read row you re-requested).

BYOK key unavailable / decryption failed — the BYOK-enforced sibling: 409 BYOK_KEY_UNAVAILABLE and the key- lifecycle cause table.
Error Code Reference — the ENCRYPTION_FAILED and DECRYPTION_FAILED rows.
Rate-limit and cooldown taxonomy — the deterministic / transient / conditional retry taxonomy this runbook’s ladder applies.
Customer-Managed Keys (BYOK) — the lifecycle and what enforce actually scopes.
SMPP credential provisioning unavailable — the SMPP issuance/rotation 503 whose checklist includes the org-encryption dependency.

​Troubleshooting: ENCRYPTION_FAILED (5xx) and DECRYPTION_FAILED (502)

​Symptom

​Reproduce and capture the handle

​Cause table

​Recovery ladder

​What NOT to do

​When to escalate

​Related