op-acceptance-tests: cover proofs history across an unclean shutdown

[einar-oplabs]

Why

After a SIGKILL, OOM or power loss, an op-reth proofs-history store usually holds blocks reth's datadir has not persisted (reth persists every --engine.persistence-threshold blocks, the store every 5). Nothing reconciles the two at startup, so a node that comes back onto a chain the sequencer rebuilt serves stale eth_getProof data and then crash-loops on ParentHashMismatch (ethereum-optimism/protocol-team#464, fixed by #23358). Nothing tested the unclean-shutdown path at all; these are the acceptance tests for it.

What

op-acceptance-tests/tests/proofs_history_restart (sequencer without proofs history as the eth_getProof reference, follower with proofs history under the cell's flags, 1s L2 blocks, both op-reths with --engine.persistence-threshold and --engine.memory-block-buffer-target=5 passed explicitly):

• TestProofsHistoryFollowerRecoversFromUncleanShutdown: traffic (transfers back and forth plus a contract that writes NUMBER/TIMESTAMP to two slots) runs until the follower's proofs store has flushed the cell's target block, the follower's op-reth is SIGKILLed at the cell's kill point and its CL stopped. On restart the test reads reth's datadir tip before the CL comes back and requires it below the store's tip (otherwise the cell is not exercising the crash gap), re-peers EL and CL, waits for the follower to be in sync with the sequencer, drives ten more blocks and sweeps VerifyHistoricalProofsMatch from the datadir tip to the last traffic block for the message passer, both EOAs and the contract's two slots.
• TestProofsHistoryFollowerServesRebuiltBlocksAfterUncleanShutdown: same, but the batcher is stopped up front and the sequencer's op-reth is SIGKILLed right after the follower's, while its own unsafe blocks above the follower's store tip are still in memory; the CLs are stopped once both processes are gone. The restarted sequencer must have its datadir tip below the store tip and at or above the block that completed the workload's funding, and re-sequences the lost heights with different contents (ReorgTriggered on the store-tip block); only then is the follower restarted. Skipped naming protocol-team#464, see below.

Matrix, as table-driven subtests: threshold {7, 50} x proofs path {index = verification-interval=0, execute = =1} x restart sync mode {CL sync, EL sync}. The kill point is paired with the threshold instead of crossing the matrix again: threshold 7 kills mid-buffer (store tip 25, head 27, so the store loses blocks of its own on top of reth's), threshold 50 kills right after the store flushed block 60 (store tip = head, the store's lead over reth is largest). Observed datadir tips at the kill: 21 (once 19 or 20) at threshold 7, 46 at threshold 50, so the store is 4 to 6 and 14 blocks ahead of reth. The sequencer's datadir tip equalled the follower's in every cell.

op-devstack seams (for #23395, #23396 and later PRs):

• sysgo.SubProcess.Kill(): SIGKILL with no interrupt, bounded wait for the exit (controlledKillWait), same pipe cleanup as Stop. A process that had already died on its own is reaped with a Sub-process exited before it could be killed warning rather than reported as killed, so a crash before the kill stays visible. OpReth.StopUngracefully() exposes it; Stop() stays graceful.

• stack.UngracefulLifecycle (StopUngracefully()), forwarded by the preset l2ELFrontend, and dsl.L2ELNode.StopUngracefully() which fails the test for a node that cannot be killed (in-process op-geth).

• sysgo.MixedSingleChainNodeSpec.L2CLOpts: per-node op-node options passed to startL2CLNode (the mixed runtime passed none before, so every follower ran CL sync). sysgo.L2CLVerifierSyncMode(mode) is the option the EL-sync cells use. Ignored for kona-node like OpRethOpts is for op-geth.

• dsl.EOA.DeployBlockRecorder(): the NUMBER/TIMESTAMP recorder contract both proofs-history packages deploy, moved out of proofs_history_match so it is not copied a second time.

Unit tests: TestSubProcessKillEndsProcessThatIgnoresInterrupts (a sh that traps INT and TERM exits at once, no interrupt is sent, the sub-process restarts afterwards), TestSubProcessKillOfExitedProcess (warns instead of claiming a kill, second kill is a no-op), TestL2CLVerifierSyncMode.

Local verification

Debug op-reth built from this branch, DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=<debug op-reth> go test -count=1 -timeout 40m -parallel 2 -v -run <Test> ./op-acceptance-tests/tests/proofs_history_restart/:

--- PASS: TestProofsHistoryFollowerRecoversFromUncleanShutdown
    threshold_7-index-cl_sync-kill_mid_buffer      (62.01s)   datadir tip 21, store tip 25, head 27; sweep 21..47
    threshold_7-execute-cl_sync-kill_mid_buffer    (57.10s)   21 / 25 / 27; sweep 21..47
    threshold_7-index-el_sync-kill_mid_buffer      (51.78s)   19 / 25 / 27; sweep 19..45
    threshold_7-execute-el_sync-kill_mid_buffer    (51.83s)   21 / 25 / 27; sweep 21..44
    threshold_50-index-cl_sync-kill_after_flush    (92.36s)   46 / 60 / 60; sweep 46..84
    threshold_50-execute-cl_sync-kill_after_flush  (91.73s)   46 / 60 / 60; sweep 46..82
    threshold_50-index-el_sync-kill_after_flush    (92.20s)   46 / 60 / 60; sweep 46..78
    threshold_50-execute-el_sync-kill_after_flush  (87.43s)   46 / 60 / 60; sweep 46..78
ok      op-acceptance-tests/tests/proofs_history_restart        329.248s

The second test, run once with the skip removed, fails in all eight cells the same way: the sequencer rebuilds the store-tip block with a different hash, the restarted follower imports the chain up to the store's tip, and its op-reth exits on the next block:

ERROR Critical task `exex` panicked: `ExEx proofs-history crashed: Parent hash mismatch at block 26: expected 0xac2827d2d59b8cccf35d05720843825354f11528a90d04d6a2c86df9264dd9ab, got 0xfa3ba001f8ce6b5b0ea4af8668481d9501a55f9c4d0da786a78c55249234c2b5`   name=proofs-follower
ERROR shutting down due to error  err="Critical task `exex` panicked: ..."
    Error Trace: op-devstack/dsl/l2_el.go:72 ... proofs_history_restart_test.go:355
    Error:       failed to determine L2BlockRef of latest, could not get payload: read tcp ...: connection reset by peer
    Messages:    block not found using block label
--- FAIL: TestProofsHistoryFollowerServesRebuiltBlocksAfterUncleanShutdown
    threshold_7-index-cl_sync-kill_mid_buffer      (55.23s)   block 26: expected 0xac2827..4dd9ab, got 0xfa3ba0..34c2b5
    threshold_7-execute-cl_sync-kill_mid_buffer    (54.78s)   block 26
    threshold_7-index-el_sync-kill_mid_buffer      (50.90s)   block 26
    threshold_7-execute-el_sync-kill_mid_buffer    (49.20s)   block 26
    threshold_50-index-cl_sync-kill_after_flush    (85.86s)   block 61: expected 0xa8efca..28e6c1, got 0x3db150..ba7192
    threshold_50-execute-cl_sync-kill_after_flush  (88.87s)   block 61
    threshold_50-index-el_sync-kill_after_flush    (80.87s)   block 61
    threshold_50-execute-el_sync-kill_after_flush  (84.00s)   block 61
FAIL    op-acceptance-tests/tests/proofs_history_restart        281.623s

"expected" is the hash of the store's tip block on the chain that was killed, "got" the rebuilt block at the same height; the ExEx had skipped the rebuilt blocks below the tip by height. The test's own failure is the sync wait losing the follower's RPC, with the panic in the log right above it.

After the review changes (both op-reths killed back to back before the CLs stop, store tip re-read right before the kill, workload setup-block precondition, recorder contract in the DSL), one cell of each was re-run: threshold_7-index-el_sync-kill_mid_buffer of the first test passed (52s, datadir tip 20, store tip 25, head 27), threshold_50-index-cl_sync-kill_after_flush of the second, un-skipped, failed the same way (Parent hash mismatch at block 61; workload setup block 16, sequencer datadir tip 46), and proofs_history_match/threshold_7-masking_0-index passed with the moved recorder helper.

go test -count=1 ./op-devstack/sysgo/ -run 'TestSubProcess|TestL2CLVerifierSyncMode'   # ok

CI budget: with -parallel 1 the eight cells of the first test take about 10 minutes back to back on this machine, and the second test the same once #23358 un-skips it, so the package stays under the 30-minute package timeout with about 10 minutes to spare. If CI boxes turn out slower, the rebuild test can move to its own package.

Not run: the kona-node CL variant (DEVSTACK_L2CL_KIND=kona-node; no kona-node binary built locally). Under kona-node the CL-sync cells skip in code, since kona-node has no CL sync mode, and the EL-sync cells run; KonaNode has no StopUngracefully yet, the tests only kill op-reth.

Skipped cells

• TestProofsHistoryFollowerServesRebuiltBlocksAfterUncleanShutdown, all cells: ethereum-optimism/protocol-team#464; op-reth: reconcile the proofs store with reth's chain at startup #23358 un-skips it.
• Both tests, CL-sync cells under DEVSTACK_L2CL_KIND=kona-node: kona-node has no CL sync mode.

Findings and deviations from the issue

• The first test passes on develop. Implement CI release pipeline #464 says the same-blocks case is harmless and the runs confirm it: the ExEx skips the re-imported blocks by height and the store's copies match. The issue expected both tests to fail; only the rebuilt-blocks one does. The first test stays unskipped as the regression test for the recovery path itself and should keep passing with op-reth: reconcile the proofs store with reth's chain at startup #23358, which re-indexes those blocks instead of skipping them.
• The issue's "about 55 blocks lost at threshold 50 after 60 blocks" is not what reth does: with buffer target 5 it persists once in-memory blocks exceed the threshold and keeps 5, so after block 51 the datadir is at 46 and stays there until block 97. The store's lead over reth after 60 blocks is 14, which reproduces Implement CI release pipeline #464 just as well; the kill height stays at 60 as the issue asks.
• The follower's CL is stopped gracefully alongside the EL kill and restarted after it (op-node runs in-process and cannot be killed). Reading reth's head between the EL and CL starts is what gives the test reth's datadir tip without a log hook.
• TestResyncing in rust/op-reth/tests/proofs/core stays as the graceful variant.
• --engine.memory-block-buffer-target is passed explicitly next to the threshold so the gap does not depend on reth's defaults (develop's reth 2.5.2 pin: threshold 7, target 5).

Stacked on #23418.

Closes #23380

🤖 Generated with Claude Code