Fix reader, poller and firehose server bugs#175
Draft
sduchesneau wants to merge 5 commits into
Draft
Conversation
fetchBlockWithHash reset optimisticallyPolledBlocks without holding optimisticallyPolledBlocksLock, racing with the batch-fetch nailer goroutine writing to the same map.
LastSeenBlockNum panicked when a backup ran before any block was seen (#90). Stopped/LastExitCode read s.cmd without cmdLock while Stop nils it under lock.
Missing return after Shutdown let the read loop start with a nil console reader, panicking in a goroutine.
Nobody shut down the FileUploader, so archiver termination completed with uploads pending and the canceled lifecycle ctx stranded final one-block files locally (#53). The final pass now runs on a fresh bounded context before termination completes. Also fix a data race on the walk callback's err captured by upload goroutines.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five independent fixes across the firehose server, block poller and node-manager, one commit each.
Fixes
fmt.Printlndebug statements from the Blocks handler error paths (blocks.go); the structured zap logging next to them is kept.fetchBlockWithHashresetoptimisticallyPolledBlockswithout holdingoptimisticallyPolledBlocksLock, racing with the batch-fetch nailer goroutine that writes the same map under lock. The reset is now locked.LastSeenBlockNum()dereferencedLastSeenBlock()without a nil check, panicking when a backup ran before any block was seen (Fixesreader-nodebackup panics if no block ever seen on the node #90).Stopped()andLastExitCode()reads.cmdwithoutcmdLockwhileStop()nils it under lock; both now take the lock.Launch()was missing areturnafterp.Shutdown(err)when the console reader factory fails, so the read loop started on a nil console reader and panicked in a goroutine.FileUploader: the archiver'sOnTerminatingonly logged, so mindreader termination completed with uploads pending, and the canceled plugin lifecycle context stranded the final one-block files locally (Fixes firehose exits before zipping all the one block files #53). Archiver termination now shuts the uploader down and waits for a final synchronous upload pass that runs on a fresh, bounded (5 min) context beforeTerminated()completes. Also fixes a data race inuploadFileswhere the walk callback's namederrreturn was written from the concurrent upload goroutines.Tests
All new tests were verified to fail (race report, panic, or assertion) with the fix reverted, and pass with it:
TestBlockPoller_FetchBlockWithHashConcurrentWithBatchFetch(blockpoller/poller_race_test.go): runsfetchBlockWithHashconcurrently withloadNextBlocks; races under-racepre-fix.TestSuperviser_LastSeenBlockNumWithoutAnyBlockSeen(node-manager/superviser/superviser_race_test.go): nil-derefs pre-fix, returns 0 post-fix.TestSuperviser_StoppedAndLastExitCodeConcurrentWithStop: exercises the exacts.cmdwrite pattern ofStart()/Stop()against concurrentStopped()/LastExitCode()readers; races under-racepre-fix. It deliberately avoids driving a real process because that trips a pre-existing, unrelated race on overseer'sCmd.State(see below).TestMindReaderPlugin_LaunchWithFailingConsoleReaderFactory(node-manager/mindreader/mindreader_launch_test.go): crashes with a goroutine panic pre-fix, terminates cleanly with the factory error post-fix.TestArchiver_ShutdownDrainsFileUploader(node-manager/mindreader/archiver_test.go): writes one-block files to a mock local store after the lifecycle context is canceled, shuts the archiver down, and asserts the files reached the destination store beforeTerminated()returns; fails pre-fix.go test -race ./firehose/... ./blockpoller/... ./node-manager/...passes except for three pre-existing failures innode-manager/superviser(TestSuperviser_StartsCorrectly,TestSuperviser_CanBeRestartedCorrectly,TestSuperviser_CapturesStdoutCorrectly) caused by a data race inside theoverseerdependency (Cmd.Statewritten by overseer's run goroutine whileisRunning()reads it): they fail identically on a cleandevelopcheckout and are untouched by this PR. Without-racethe whole superviser package passes.Fixes #90
Fixes #53
🤖 Generated with Claude Code