Skip to content

Fix reader, poller and firehose server bugs#175

Draft
sduchesneau wants to merge 5 commits into
developfrom
fix/reader-poller-server-bugs
Draft

Fix reader, poller and firehose server bugs#175
sduchesneau wants to merge 5 commits into
developfrom
fix/reader-poller-server-bugs

Conversation

@sduchesneau

Copy link
Copy Markdown
Contributor

Five independent fixes across the firehose server, block poller and node-manager, one commit each.

Fixes

  • firehose/server: remove six leftover fmt.Println debug statements from the Blocks handler error paths (blocks.go); the structured zap logging next to them is kept.
  • blockpoller: fetchBlockWithHash reset optimisticallyPolledBlocks without holding optimisticallyPolledBlocksLock, racing with the batch-fetch nailer goroutine that writes the same map under lock. The reset is now locked.
  • node-manager/superviser: LastSeenBlockNum() dereferenced LastSeenBlock() without a nil check, panicking when a backup ran before any block was seen (Fixes reader-node backup panics if no block ever seen on the node #90). Stopped() and LastExitCode() read s.cmd without cmdLock while Stop() nils it under lock; both now take the lock.
  • node-manager/mindreader: Launch() was missing a return after p.Shutdown(err) when the console reader factory fails, so the read loop started on a nil console reader and panicked in a goroutine.
  • node-manager/mindreader: nothing ever shut down the FileUploader: the archiver's OnTerminating only logged, so mindreader termination completed with uploads pending, and the canceled plugin lifecycle context stranded the final one-block files locally (Fixes firehose exits before zipping all the one block files #53). Archiver termination now shuts the uploader down and waits for a final synchronous upload pass that runs on a fresh, bounded (5 min) context before Terminated() completes. Also fixes a data race in uploadFiles where the walk callback's named err return was written from the concurrent upload goroutines.

Tests

All new tests were verified to fail (race report, panic, or assertion) with the fix reverted, and pass with it:

  • TestBlockPoller_FetchBlockWithHashConcurrentWithBatchFetch (blockpoller/poller_race_test.go): runs fetchBlockWithHash concurrently with loadNextBlocks; races under -race pre-fix.
  • TestSuperviser_LastSeenBlockNumWithoutAnyBlockSeen (node-manager/superviser/superviser_race_test.go): nil-derefs pre-fix, returns 0 post-fix.
  • TestSuperviser_StoppedAndLastExitCodeConcurrentWithStop: exercises the exact s.cmd write pattern of Start()/Stop() against concurrent Stopped()/LastExitCode() readers; races under -race pre-fix. It deliberately avoids driving a real process because that trips a pre-existing, unrelated race on overseer's Cmd.State (see below).
  • TestMindReaderPlugin_LaunchWithFailingConsoleReaderFactory (node-manager/mindreader/mindreader_launch_test.go): crashes with a goroutine panic pre-fix, terminates cleanly with the factory error post-fix.
  • TestArchiver_ShutdownDrainsFileUploader (node-manager/mindreader/archiver_test.go): writes one-block files to a mock local store after the lifecycle context is canceled, shuts the archiver down, and asserts the files reached the destination store before Terminated() returns; fails pre-fix.

go test -race ./firehose/... ./blockpoller/... ./node-manager/... passes except for three pre-existing failures in node-manager/superviser (TestSuperviser_StartsCorrectly, TestSuperviser_CanBeRestartedCorrectly, TestSuperviser_CapturesStdoutCorrectly) caused by a data race inside the overseer dependency (Cmd.State written by overseer's run goroutine while isRunning() reads it): they fail identically on a clean develop checkout and are untouched by this PR. Without -race the whole superviser package passes.

Fixes #90
Fixes #53

🤖 Generated with Claude Code

fetchBlockWithHash reset optimisticallyPolledBlocks without
holding optimisticallyPolledBlocksLock, racing with the
batch-fetch nailer goroutine writing to the same map.
LastSeenBlockNum panicked when a backup ran before any block
was seen (#90). Stopped/LastExitCode read s.cmd without
cmdLock while Stop nils it under lock.
Missing return after Shutdown let the read loop start with a
nil console reader, panicking in a goroutine.
Nobody shut down the FileUploader, so archiver termination
completed with uploads pending and the canceled lifecycle ctx
stranded final one-block files locally (#53). The final pass
now runs on a fresh bounded context before termination
completes. Also fix a data race on the walk callback's err
captured by upload goroutines.
@sduchesneau

sduchesneau commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

🔍 Vulnerabilities of ghcr.io/streamingfast/firehose-core:0b9cb29-amd64

📦 Image Reference ghcr.io/streamingfast/firehose-core:0b9cb29-amd64
digestsha256:4ce1b90b4cfbc4dd68f455bef6f5bdb7b07b5c6d5704896a655fe12dae8703c6
vulnerabilitiescritical: 0 high: 0 medium: 0 low: 0
platformlinux/amd64
size162 MB
packages506
📦 Base Image ubuntu:24.04
also known as
  • noble
  • noble-20260610
digestsha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf
vulnerabilitiescritical: 0 high: 0 medium: 12 low: 5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reader-node backup panics if no block ever seen on the node firehose exits before zipping all the one block files

1 participant