Troubleshooting: idempotency, balance re-entry, and auto-top-up gates

Money-moving requests carry extra guards beyond the normal error contract: idempotency keys that must be present and well-formed, a wallet-level re-entry window that refuses concurrent debits under one key, and an auto-top-up scheduler that stops charging you when its own caps or the card stop cooperating. This page maps each code to its cause and its recovery so a 400/409 from one of these gates is a five-minute fix, not a support ticket.

This page covers money-mutation guards. If outbound sends are failing with 402, work the billing-gate outbound blocks page instead; the full idempotency contract and SDK key generation are in Idempotency and safe retries.

1. Idempotency key gates — required, malformed, reused

POST /api/v1/billing/balance/top-up (alias POST /api/v1/billing/credits/purchase) rejects a request with no Idempotency-Key header:

Code	HTTP	Cause	Recovery
`IDEMPOTENCY_KEY_REQUIRED`	400	The header was omitted entirely	Add `Idempotency-Key: <stable client-generated value>` to every `POST /balance/top-up` call and reuse the same key on retries
`INVALID_IDEMPOTENCY_KEY`	400	The key was present but failed shape validation (under 8 chars, over 255, or whitespace-padded)	Generate an 8–255 character string — a UUID works; trim whitespace before sending
`IDEMPOTENCY_KEY_REUSED`	409	The same key was replayed with a different request body, and the first attempt succeeded	Mint a fresh key for the genuinely different request; reuse the key only for byte-identical retries

The same contract protects every POST/PUT that accepts the header: the fingerprint of the body is stored with the cached response, so replaying the key with a changed payload returns 409 instead of silently replaying the earlier answer. One escape hatch exists on purpose — if the first attempt failed (4xx/5xx), the same key with a corrected body is allowed through and runs as a fresh request, so a fixed payload is never trapped in a 409 loop. The balance/top-up endpoint answers a replay with the original Stripe checkout URL rather than minting a second session: a Redis cache covers back-to-back double-clicks, and a durable record covers the 24-hour replay window.

2. DEDUCT_IN_FLIGHT — the balance re-entry window

DEDUCT_IN_FLIGHT (409) comes from the wallet layer itself (deductBalance / addBalance), not the API surface. It means another request is already executing a balance mutation under the same idempotency key and has not posted its result yet. The wallet claims the key in Redis; when the claim fails because the saved value is still pending, it polls briefly (about two seconds) for the winner’s result and then throws this error instead of reclaiming the key — reclaiming would risk a double-debit. Recovery:

Retry the exact same key after a short backoff (1–3 seconds). Three outcomes, all safe: the original finished and you get its cached result (no second debit); it is still running and you get the same 409 again — back off and repeat; or the original crashed and the 24-hour key TTL eventually expires, letting a later retry proceed fresh.
Randomize key collisions away. If this fires under normal load, your key generator is colliding — one key per logical operation, not one key per batch or per customer.
Never treat it as a terminal failure. The 409 is asking for a poll, not a fix; the final state is decided when the in-flight caller posts its result.

The same retry-safe rule covers the sibling concurrency-in-flight guards: a 409 CONFLICT on the general Idempotency-Key plugin means “a request with this key is in progress; retry shortly to receive its result” and carries a Retry-After: 2 header.

3. Auto-top-up — cap reached and payment failed

Auto-top-up runs on the scheduler: when the live balance drops below your configured threshold_minor, it charges the org’s default card off-session for the recharge_amount_minor and credits the wallet. Two stops surface as notifications and audit entries rather than as request errors:

Code	Where you see it	Cause	Recovery
`AUTO_TOPUP_MONTHLY_CAP_REACHED`	Dashboard notification / audit	The calendar-month total charged would exceed your `max_monthly_minor` cap; the scheduler skips the org for the rest of the month	Raise the cap (`PUT /api/v1/billing/auto-topup` with a higher `max_monthly_minor`), or top up manually until the month rolls over (UTC); if the cap blocks the very first charge of the month, it is also below `recharge_amount_minor` — set it to at least one recharge
`AUTO_TOPUP_PAYMENT_FAILED`	Dashboard notification / audit	The off-session Stripe PaymentIntent failed — card declined, expired, 3DS required, or no default payment method on the customer	Update the card in Billing → Payment methods (or run one manual `POST /balance/top-up` so a fresh default method is set); the scheduler retries on its next tick once the balance is still under threshold and its cooldown has passed

The scheduler also enforces two tenant-side rate guards that are not reported as error codes but look similar in logs: a per-day charge cap (rolls at UTC midnight) and a one-charge-per-hour limit after the five-minute cooldown. If auto-top-up stopped without either code above, check for those two before anything else.

PAYMENT_IN_FLIGHT on crypto top-ups

POST /api/v1/billing/crypto-topup/:intentId/cancel returns 409 PAYMENT_IN_FLIGHT when the intent has already moved past waiting/confirming — the funds are credited, settling on-chain, or already refunded/failed/expired. The response’s details.current_status tells you which. If funds were sent anyway, the webhook handler credits the balance automatically even after a cancel, so the state converges on its own; refunds from a terminal state go through support, not the cancel endpoint. For card (Stripe) top-ups the analogue is the confirm flow: after POST /balance/top-up returns a checkout URL, the session is confirmed on-return by the dashboard (reconciling the same Stripe session against the idempotency record). If you bailed out mid-checkout and a later attempt reports the attempt already in motion, re-open the original checkout URL — replaying the same Idempotency-Key returns exactly that URL instead of creating a second session.

4. End-to-end example — top up safely

# 1. Generate one key per logical top-up; reuse it on every retry.
KEY="topup-$(uuidgen)"

# 2. Attempt the top-up.
curl -X POST https://orbit.devotel.io/api/v1/billing/balance/top-up \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  -d '{"amount_cents": 5000, "success_url": "https://your.app/billing/ok", "cancel_url": "https://your.app/billing/cancel"}'

# 3. On 409 DEDUCT_IN_FLIGHT / CONFLICT — wait and replay the SAME key,
#    the same body. The cached answer (the checkout URL) comes back.
sleep 2
curl -X POST https://orbit.devotel.io/api/v1/billing/balance/top-up \
  -H "X-API-Key: $ORBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  -d '{"amount_cents": 5000, "success_url": "https://your.app/billing/ok", "cancel_url": "https://your.app/billing/cancel"}'
# → 200 with an Idempotency-Replay: true header and the original checkout URL.

# 4. If the response was 400 INVALID_IDEMPOTENCY_KEY, your key failed
#    shape validation — mint a fresh 8–255 char key and retry, rather
#    than re-sending the malformed one.

5. When to escalate

Open a support ticket when the same Idempotency-Key returns DEDUCT_IN_FLIGHT for more than an hour (the 24-hour TTL self-heals crashed writers, so anything beyond that is not the normal window), or when AUTO_TOPUP_PAYMENT_FAILED keeps firing after the default payment method is known-good. Include:

Your organization ID (dashboard → Settings, or GET /api/v1/me as organizationId).
The exact error code, the Idempotency-Key you sent, and the request ID from the error envelope.
For auto-top-up, a timestamp of the last successful charge (visible in Billing → Auto-top-up).

Reference

Troubleshooting: idempotency, balance re-entry, and auto-top-up gates

Troubleshooting: idempotency, balance re-entry, and auto-top-up gates

1. Idempotency key gates — required, malformed, reused

2. DEDUCT_IN_FLIGHT — the balance re-entry window

3. Auto-top-up — cap reached and payment failed

PAYMENT_IN_FLIGHT on crypto top-ups

4. End-to-end example — top up safely

5. When to escalate

See also

​Troubleshooting: idempotency, balance re-entry, and auto-top-up gates

​1. Idempotency key gates — required, malformed, reused

​2. DEDUCT_IN_FLIGHT — the balance re-entry window

​3. Auto-top-up — cap reached and payment failed

​PAYMENT_IN_FLIGHT on crypto top-ups

​4. End-to-end example — top up safely

​5. When to escalate

​See also

Troubleshooting: idempotency, balance re-entry, and auto-top-up gates

1. Idempotency key gates — required, malformed, reused

2. DEDUCT_IN_FLIGHT — the balance re-entry window

3. Auto-top-up — cap reached and payment failed

PAYMENT_IN_FLIGHT on crypto top-ups

4. End-to-end example — top up safely

5. When to escalate

See also