fix(core/txpool): stop delivering special tx events under the pool lock - #2519
Open
gzliudan wants to merge 1 commit into
Open
fix(core/txpool): stop delivering special tx events under the pool lock#2519gzliudan wants to merge 1 commit into
gzliudan wants to merge 1 commit into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
gzliudan
requested review from
AnilChinchawale,
anunay-xin,
benjamin202410,
liam-lai and
wanwiset25
August 12, 2026 23:09
benjamin202410
approved these changes
Aug 13, 2026
gzliudan
force-pushed
the
fix-special-tx-feed-under-lock
branch
3 times, most recently
from
August 16, 2026 14:06
2e75b6c to
41160b4
Compare
promoteSpecialTx called txFeed.Send while holding pool.mu, so a subscriber that stopped draining froze the whole pool. On mainnet a peer lost its transaction broadcaster, AsyncSendTransactions blocked, txBroadcastLoop stopped draining pm.txsCh, and the resulting Feed.Send pinned pool.mu for hours: 131 goroutines piled up in Add, 20 in Pending, runReorg never ran and the node stopped importing blocks while RPC stayed responsive. Route the event through queueTxEvent like every other path in add(), so runReorg delivers it after releasing the lock. Add TestSpecialTxPromotionDoesNotBlockOnTxFeed. It subscribes twice, with a channel that is never read and one that is, so it pins down both invariants: adding a special tx must not block behind a stalled subscriber, and the event must still reach the subscribers that are keeping up. This fixes the pool-side amplifier only. The peer-side defect that starts the failure chain (eth/peer.go: broadcastTransactions can exit while the peer stays alive, so AsyncSendTransactions blocks forever) is tracked separately. With a stalled subscriber, a full pm.txsCh can now block only the reorg goroutine instead of every Add. Delivery timing changes slightly: for remote adds the special tx event is now sent after the next reorg run instead of synchronously inside Add, matching all other transactions. Local adds still deliver before Add returns.
gzliudan
force-pushed
the
fix-special-tx-feed-under-lock
branch
from
August 16, 2026 14:43
41160b4 to
d7614e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Problem
A mainnet node froze for ~2h40m while RPC stayed responsive: no new blocks were imported.
debug.stacks()showed the full failure chain:broadcastTransactionsexited while the peer stayed alive).eth/peer.goAsyncSendTransactionsblocked forever onp.txBroadcast, which no longer had a reader.txBroadcastLoopstopped drainingpm.txsChand became a stalledtxFeedsubscriber.LegacyPool.promoteSpecialTxcalledpool.txFeed.Sendwhile holdingpool.mu, soFeed.Sendpinned the pool write lock for hours.Result: 131 goroutines piled up in
LegacyPool.Add, 20 inPending,runReorgnever ran, and the node stopped importing blocks.Fix
Route the special tx event through
queueTxEvent, the same path every other transaction inadd()uses. The event is delivered byrunReorgafterpool.muis released, so a stalled subscriber can no longer pin the pool lock. This also aligns the XDPoS-specific path with the upstream geth design.Behavior change
Remote adds (
sync=false) now deliver theNewTxsEventfor special txs after the next reorg run instead of synchronously insideAdd— identical to normal transactions. Local adds (sync=true) still deliver beforeAddreturns. No consensus, RPC, or API surface changes.Scope / follow-up
This fixes the pool-side amplifier only. The peer-side defect that starts the chain (
eth/peer.go:broadcastTransactionscan exit while the peer stays alive, leavingAsyncSendTransactionsblocked forever) is tracked separately and needs its own fix. Until then, a stalled subscriber plus a fullpm.txsChcan block the reorg goroutine, but no longerAdd/Pending.Types of changes
What types of changes does your code introduce to XDC network?
Put an
✅in the boxes that applyImpacted Components
Which parts of the codebase does this PR touch?
Put an
✅in the boxes that applyChecklist
Put an
✅in the boxes once you have confirmed below actions (or provide reasons on not doing so) that