fix(api): auto-recover Axelar stuck-confirm and stop leaked squidRouterPay polling loops#1276
Merged
Merged
Conversation
…erPay polling loops
Two fixes for the squidRouterPay phase, prompted by a Base->BNB transfer that
sat in Axelar status "called" for 14h after its validator confirmation poll
failed (Axelar's relayer never retries a failed poll):
- Honor the phase processor's AbortSignal in the status polling loop and
balance check. Previously every timed-out execution left an immortal 10s
polling loop behind; retries plus the recovery worker piled up dozens of
them per stuck ramp (~336 log lines/2min observed in production) until the
SquidRouter status API rate-limited us with 429s.
- Detect a failed confirmation poll (status "called" + confirm_failed from
axelarscan) and auto-recover: fetch a signed ConfirmGatewayTx from Axelar's
public recovery signing service and broadcast it to the Axelar RPC, which
restarts the validator poll. Uses only the public tx hash - no keys, no
funds. Attempts are rate-limited via a cooldown timestamp persisted in ramp
state. The byte handling is done manually because axelarjs-sdk's
manualRelayToDestChain mangles the relayer's numeric-keyed byte response
and broadcasts an empty tx ("must contain at least one message").
✅ Deploy Preview for vortex-sandbox ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for vortexfi ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for vrtx-dashboard canceled.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens SquidRouter/Axelar on-ramp processing after a production incident by (1) ensuring phase executions honor the processor’s AbortSignal to prevent leaked polling loops, and (2) adding an automated recovery path for Axelar GMP transfers stuck at status: "called" with confirm_failed by re-broadcasting a ConfirmGatewayTx via Axelar’s public recovery relayer + RPC.
Changes:
- Add
recoverAxelarStuckConfirmin@vortexfi/sharedand extend AxelarScan status typing to surfaceconfirm_failed+ source chain metadata. - Thread
AbortSignalthrough initial delay/polling inSquidRouterPayPhaseHandler, and implement rate-limited stuck-confirm recovery persisted viaaxelarConfirmRecoveryAt. - Add unit tests for relayer byte decoding/broadcast behavior and API handler integration tests (including an abort regression test), plus update the security spec.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/shared/src/services/squidrouter/axelar.ts | Exposes confirm_failed info and adds recoverAxelarStuckConfirm to broadcast a relayer-signed ConfirmGatewayTx. |
| packages/shared/src/services/squidrouter/axelar.test.ts | Tests relayer byte-shape decoding and broadcast error handling for recovery. |
| apps/api/src/api/services/phases/meta-state-types.ts | Adds axelarConfirmRecoveryAt to persist recovery cooldown timestamp in ramp state metadata. |
| apps/api/src/api/services/phases/handlers/squid-router-pay-phase-handler.ts | Honors AbortSignal for sleeps, triggers rate-limited stuck-confirm recovery, and persists recovery attempt timestamps. |
| apps/api/src/api/services/phases/handlers/squid-router-pay-phase-handler.test.ts | Verifies recovery trigger/cooldown and asserts polling stops after abort. |
| docs/security-spec/05-integrations/squid-router.md | Documents abort-aware polling and the new Axelar stuck-confirm recovery failure mode + mitigation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Addresses Copilot review on #1276: - recoverAxelarStuckConfirm: validate the relayer response shape (reject null/non-object data, ignore non-numeric keys, require integer bytes in 0-255) instead of silently coercing garbage into a corrupted broadcast. - Treat a successful broadcast without a returned tx hash as an error rather than returning an empty string. - Thread the phase AbortSignal into the recovery fetches so aborted executions stop promptly instead of finishing network I/O after timeout. - Normalize an unparseable persisted axelarConfirmRecoveryAt to "never attempted" so the cooldown comparison stays well-defined.
Addresses Copilot follow-up on #1276: byte-by-byte string concatenation before btoa is quadratic; encode in 8KiB String.fromCharCode chunks instead (spreading the full array at once could hit the argument-count limit).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
A Base → BNB onramp transfer (
0x31365ff4…236f49, rampe6f7eb01-…) sat in Axelar statuscalledfor ~14 hours because its validator confirmation poll failed — and Axelar's relayer never retries a failed poll. It was recovered manually by broadcasting a freshConfirmGatewayTx. This PR automates that recovery and fixes the retry storm the incident exposed in production logs.Fix 1: honor the processor's
AbortSignal(retry-storm / 429s)PhaseProcessortimes out a phase execution after 10 minutes and aborts it, butSquidRouterPayPhaseHandlerignored the signal: every timed-out execution left an immortal 10s polling loop behind. Retries (8×) plus the every-5-min recovery worker kept spawning executions, each leaking another loop — production showed ~336 handler log lines per 2-minute window (one healthy loop produces ~12–24) andHTTP 429rate-limiting from the SquidRouter status API.The handler now threads the signal through the initial delay, the status polling loop, and the destination balance check (via the existing signal-aware
sleep), so abandoned executions unwind immediately.Fix 2: auto-recover a failed confirmation poll
When axelarscan reports
status: "called"withconfirm_failed, the handler now calls the newrecoverAxelarStuckConfirm(shared): fetch a signedConfirmGatewayTxfrom Axelar's public recovery signing service and broadcast it to the Axelar RPC, restarting the validator poll. Notes:axelarConfirmRecoveryAt), so retried/concurrent executions don't re-broadcast.axelarjs-sdk'smanualRelayToDestChainmangles the relayer's numeric-keyed JSON byte response and broadcasts an empty tx (must contain at least one message); verified working against the real services during the incident.Tests
packages/sharedaxelar.test.ts: numeric-keyed + array byte decode, empty-tx rejection, rejected-broadcast rejection.apps/apisquid-router-pay-phase-handler.test.ts: recovery triggered with timestamp persisted, cooldown respected, no recovery on healthy statuses, and a regression test asserting polling stops after abort.Security spec updated:
docs/security-spec/05-integrations/squid-router.md(handler behavior + new failure-mode row).Verification
bun typecheckclean across the monorepo, Biome clean,bun build:sharedrebuilt