SIGSEGV in red-black tree rebalance during block connect (v9.26.5) — reproduced on multiple independent nodes simultaneously
Summary
On 2026-08-27 at ~04:37 UTC, digibyted v9.26.5 crashed with SIGSEGV on the b-msghand thread while connecting a new block. At least three independent nodes went down within ~90 seconds of each other, each at a different block height.
A core dump was captured. The fault is a null pointer dereference in a red-black tree rebalancing loop: a child pointer is validated, a rotation is called that overwrites that same field, and the caller reloads the field without re-checking it for null.
The chain itself was unaffected — block production held steady at ~240 blocks/hr throughout.
Environment
|
|
| Version |
DigiByte version v9.26.5 (release build) |
| Platform |
x86-64, Linux 7.0.0-30-generic, Docker |
| Relevant config |
digidollar=1, digidollarstatsindex=1, txindex=1, dbcache=1000, v2transport=1 |
| Role |
Mining node serving getblocktemplate to a stratum pool (~15s poll) |
Impact
- Node terminates. With
restart: unless-stopped it recovers; without auto-restart it stays down.
- Unclean exit forces a full index rebuild from height 0. After restart:
ERROR: Commit: Failed to commit latest txindex state
ERROR: Commit: Failed to commit latest digidollarstatsindex state
Syncing txindex with block chain from height 0
Syncing digidollarstatsindex with block chain from height 0
txindex took ~25 min to rebuild, digidollarstatsindex ~36 min. getrawtransaction is degraded throughout.
- During the replay window the node serves stale block templates.
IsInitialBlockDownload() returns false (tip timestamp is recent and nMinimumChainWork=0), and progress=0.999996, so getblocktemplate happily builds on a tip ~140 blocks behind the network. A pool mining against it produced a block that was rejected as inconclusive. Arguably getblocktemplate should refuse to serve while headers - blocks is large.
Crash details
Kernel:
b-msghand[3331]: segfault at 8 ip 000059159794ab40 sp 00007fce87ff5c70 error 4
in digibyted[4c5b40,591597521000+f16000]
From the core dump (43 threads captured):
SIGINFO: signo=11 (SIGSEGV) code=1 (SEGV_MAPERR) si_addr = 0x8
RIP = digibyted+0x4c5b40
RDI = 0x0 <-- null pointer in first-arg / `this` register
RAX = RBX = RDX = 0
RSI = RBP = 0x59159d6dd3f0 (self-referential node = empty container sentinel)
Only 1 of 43 threads was executing DigiByte code at the moment of the crash; the other 42 were parked in libc (futex/epoll/syscalls).
Root cause
Disassembly of the stripped release binary (verified as the exact crashed build — first-4KB sha256 and all four LOAD segments match the core):
4c5b31: mov 0x10(%rbp),%rdi ; rdi = node->right
4c5b35: mov (%rdi),%rax ; dereference SUCCEEDS -> rdi was NON-null here
4c5b38: test $0x1,%al ; mark bit set?
4c5b3a: je 4c5cd8 ; not marked -> go mark it
...
4c5cd8: or $0x1,%rax
4c5cdc: mov %rax,(%rdi) ; set mark bit
4c5ce2: andq $0xfffffffffffffffe,0x0(%rbp)
4c5ce7: call 4c3910 ; <-- rotation, see below
4c5cec: mov 0x10(%rbp),%rdi ; <-- RELOAD of the same field, NO null re-check
4c5cf0: jmp 4c5b40
...
4c5b40: mov 0x8(%rdi),%rdx ; *** SIGSEGV: rdi = 0, faulting address 0x8 ***
Note 0x4c5b40 is reached by a backward jump from 0x4c5cf0, not by falling through from 0x4c5b35. So the pointer was non-null on first load and null on reload; the only thing in between is the call.
The callee at 0x4c3910 is a left-rotation, and it writes the field the caller reloads:
4c3922: mov 0x10(%rdi),%rax ; rax = node->right
4c3926: mov 0x8(%rax),%rdx ; rdx = rax->left
4c392a: mov %rdx,0x10(%rdi) ; node->right = rdx <-- OVERWRITES [node+0x10]
4c392e: test %rdx,%rdx ; <-- the rotation DOES null-check this value
4c3931: je 4c393f
The rotation null-checks the value it stores. The caller, after reloading the same field, does not.
Node layout is parent-pointer-with-color-tag at +0x0, left at +0x8, right at +0x10 — a compact intrusive red-black tree (boost::intrusive / multi_index style), not libstdc++ std::map (color@0, parent@8, left@0x10, right@0x18).
Two possible readings
(a) Missing null re-check in the rebalance loop after the rotation.
(b) The tree was already corrupted on entry, e.g. mutated without synchronization, and this loop is simply where the damage surfaces.
Boost's rebalancing code is mature, and the asymmetry above (careful callee, careless caller) is often a sign the caller was written against an invariant something else violated. (b) deserves weight. An unstripped build would settle it immediately:
addr2line -f -C -e digibyted-with-symbols 0x4c5b40 0x4c3910
Log context
The last three lines before the crash, and the pattern they break:
04:37:35Z Saw new header hash=7fc40d8b...c8 height=24104109
04:37:35Z Oracle: Price cache updated for height 24104109: 5012 micro-USD, source_time=1787804919
04:37:35Z Oracle: Manually cleared all pending messages and attestations
<SIGSEGV>
The normal sequence — observed 3,964 times in the same log — is four lines:
Oracle: Price cache updated for height N
Oracle: Manually cleared all pending messages and attestations
UpdateTip: new best=... height=N <-- never reached
Oracle: AddOracleBundleToBlock ... height N+1
A clear/erase immediately preceding a rebalance crash is consistent with the erase path driving the rotations.
52 seconds earlier there was a 1-block reorg at height 24104107. Six other reorgs in the same log were handled without incident, so a reorg alone is not sufficient — but it perturbs oracle/DigiDollar state and may widen the window.
Evidence this is not bad block data
- The block that crashed the node,
7fc40d8b... (height 24104109, 5 txs, qubit), reconnected cleanly on retry after restart and is in the canonical chain.
- Three nodes died at three different heights within ~90s: one operator's node last reported 24104106; a second node on a different host was already refusing connections by 04:36:53; the node with the core dump died at 24104109. Deterministic bad input would kill every node at the same block.
- No validation rejections anywhere in the log: zero
Misbehaving, zero bad-txns, zero bad-blk, zero bans.
- Block timestamp anomalies were ruled out: the +250s forward jump near the crash does not even rank in the top six such jumps across 4,308 blocks in the same log, and the larger ones were harmless.
Different nodes hold different pending oracle message/attestation sets, which is a plausible explanation for different crash heights without requiring a thread race — though (b) above would also explain it.
Secondary issue: unchecked FlatFilePos in the getblocktemplate path
Likely the same missing-null-check family, still occurring on a healthy node:
08:06:52Z Oracle: Added MuSig2 v0x03 bundle to block 24104959 (epoch 602623)
08:06:52Z CreateNewBlock(): block weight: 3764 txs: 2 fees: 200000 sigops 416
08:06:52Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for FlatFilePos(nFile=-1, nPos=0)
nFile=-1, nPos=0 is a default-constructed FlatFilePos — a "not found" lookup handed to ReadBlockFromDisk unchecked. It fires immediately after CreateNewBlock(), and the error count tracks specific unconfirmed transactions, not template size: 43-tx templates are often clean while 3-tx templates can error. A run of 4 errors/template began when transactions entered the mempool at 02:34:07 and stopped the instant block 24103628 confirmed them at 02:34:43. Harmless today (logged and continues), but remotely reachable by anyone who can broadcast a transaction.
Available on request
- Full 8.8 GB core dump (43 threads)
debug.log covering ~17h around the event- Register state for all 43 threads
- Candidate stack return addresses (raw scan; build has no frame pointers)
Suggested next steps
addr2line the two offsets against an unstripped v9.26.5 build to name the function and container.- Determine whether the container mutated during the traversal is accessed from more than one thread without a lock.
- Consider a null re-check after rotation in the rebalance loop regardless of root cause.
- Add a null check before
ReadBlockFromDisk for default-constructed FlatFilePos.
- Consider having
getblocktemplate refuse to serve while headers - blocks exceeds a small threshold, so pools cannot mine on a replaying node.
SIGSEGV in red-black tree rebalance during block connect (v9.26.5) — reproduced on multiple independent nodes simultaneously
Summary
On 2026-08-27 at ~04:37 UTC,
digibytedv9.26.5 crashed withSIGSEGVon theb-msghandthread while connecting a new block. At least three independent nodes went down within ~90 seconds of each other, each at a different block height.A core dump was captured. The fault is a null pointer dereference in a red-black tree rebalancing loop: a child pointer is validated, a rotation is called that overwrites that same field, and the caller reloads the field without re-checking it for null.
The chain itself was unaffected — block production held steady at ~240 blocks/hr throughout.
Environment
DigiByte version v9.26.5 (release build)digidollar=1,digidollarstatsindex=1,txindex=1,dbcache=1000,v2transport=1getblocktemplateto a stratum pool (~15s poll)Impact
restart: unless-stoppedit recovers; without auto-restart it stays down.txindextook ~25 min to rebuild,digidollarstatsindex~36 min.getrawtransactionis degraded throughout.IsInitialBlockDownload()returns false (tip timestamp is recent andnMinimumChainWork=0), andprogress=0.999996, sogetblocktemplatehappily builds on a tip ~140 blocks behind the network. A pool mining against it produced a block that was rejected asinconclusive. Arguablygetblocktemplateshould refuse to serve whileheaders - blocksis large.Crash details
Kernel:
From the core dump (43 threads captured):
Only 1 of 43 threads was executing DigiByte code at the moment of the crash; the other 42 were parked in libc (futex/epoll/syscalls).
Root cause
Disassembly of the stripped release binary (verified as the exact crashed build — first-4KB sha256 and all four LOAD segments match the core):
Note
0x4c5b40is reached by a backward jump from0x4c5cf0, not by falling through from0x4c5b35. So the pointer was non-null on first load and null on reload; the only thing in between is the call.The callee at
0x4c3910is a left-rotation, and it writes the field the caller reloads:The rotation null-checks the value it stores. The caller, after reloading the same field, does not.
Node layout is parent-pointer-with-color-tag at
+0x0, left at+0x8, right at+0x10— a compact intrusive red-black tree (boost::intrusive / multi_index style), not libstdc++std::map(color@0, parent@8, left@0x10, right@0x18).Two possible readings
(a) Missing null re-check in the rebalance loop after the rotation.
(b) The tree was already corrupted on entry, e.g. mutated without synchronization, and this loop is simply where the damage surfaces.
Boost's rebalancing code is mature, and the asymmetry above (careful callee, careless caller) is often a sign the caller was written against an invariant something else violated. (b) deserves weight. An unstripped build would settle it immediately:
Log context
The last three lines before the crash, and the pattern they break:
The normal sequence — observed 3,964 times in the same log — is four lines:
A clear/erase immediately preceding a rebalance crash is consistent with the erase path driving the rotations.
52 seconds earlier there was a 1-block reorg at height 24104107. Six other reorgs in the same log were handled without incident, so a reorg alone is not sufficient — but it perturbs oracle/DigiDollar state and may widen the window.
Evidence this is not bad block data
7fc40d8b...(height 24104109, 5 txs, qubit), reconnected cleanly on retry after restart and is in the canonical chain.Misbehaving, zerobad-txns, zerobad-blk, zero bans.Different nodes hold different pending oracle message/attestation sets, which is a plausible explanation for different crash heights without requiring a thread race — though (b) above would also explain it.
Secondary issue: unchecked
FlatFilePosin thegetblocktemplatepathLikely the same missing-null-check family, still occurring on a healthy node:
nFile=-1, nPos=0is a default-constructedFlatFilePos— a "not found" lookup handed toReadBlockFromDiskunchecked. It fires immediately afterCreateNewBlock(), and the error count tracks specific unconfirmed transactions, not template size: 43-tx templates are often clean while 3-tx templates can error. A run of 4 errors/template began when transactions entered the mempool at 02:34:07 and stopped the instant block 24103628 confirmed them at 02:34:43. Harmless today (logged and continues), but remotely reachable by anyone who can broadcast a transaction.Available on request
debug.logcovering ~17h around the eventSuggested next steps
addr2linethe two offsets against an unstripped v9.26.5 build to name the function and container.ReadBlockFromDiskfor default-constructedFlatFilePos.getblocktemplaterefuse to serve whileheaders - blocksexceeds a small threshold, so pools cannot mine on a replaying node.