op-reth: acceptance test for a sequencer EL unclean shutdown #23396

What: An acceptance test that kills the sequencer's op-reth with unsafe blocks in memory and checks the sequencer resumes from its persisted head, the followers reorg to the rebuilt chain, and the reorg depth stays within the threshold plus buffer.
Why: The bump turns the sequencer's crash gap from about 12 blocks to about 55. Nothing kills a sequencer EL today (every restart test is graceful), and the unsafe reorg every follower and the batcher must absorb becomes seven times deeper.

Description

Mixed runtime: op-reth sequencer, one op-reth follower, one op-geth follower, batcher running.

1. Produce at least 60 blocks with transactions, so the kill lands with the maximum gap.
2. Kill the sequencer EL with the kill seam from op-reth: acceptance test for proofs history across an unclean shutdown #23380. Record its head H before the kill and reth's head P right after the restart, before op-node resumes (the DSL start method must expose the EL head before the CL reconnects, or read it from the restart log).
3. Assert op-node resumes sequencing from P rather than H; both followers reorg to the rebuilt chain (ReorgTriggered / ReorgExact) within a bounded time; H - P is at most threshold plus --engine.memory-block-buffer-target; the batcher keeps submitting and the safe head advances past H; and the proof oracle from op-reth: add a windowed proof oracle to the devstack DSL #23379 on the op-reth follower matches the op-geth follower from P - 5 to the new head.

Second test: the same with the batcher stopped before the kill, so none of the lost blocks was safe, then the batcher restarted.

Matrix for this test

Threshold {7, 50} x CL {op-node, kona-node} x batcher {running, stopped}.

Verification criterion

• Passes on develop and on rust!: update reth to v2.6.0 #23046, with the measured gap reported per cell in the PR.
• A cell where the sequencer resumes from a stale H, or a follower never reorgs, is a finding against the bump; record it rather than loosening the bound.

Shared context for whoever picks this up

• The bump. rust!: update reth to v2.6.0 #23046 (reth v2.6.0), tracked in ethereum-optimism/protocol-team#354. The PR description lists the migration notes: changed defaults, rejected flags and the upstream behaviour changes. The persistence side is explained in the internal write-up Proofs exex, state masking, and persistence.
• What changes under op-reth with no diff in our tree. reth's persistence threshold default moves from 7 to 50 (now an in-memory block count; backpressure 16 to 100), so an unclean stop loses up to the threshold plus --engine.memory-block-buffer-target blocks, about 55 instead of about 12. State masking exists in reth 2.6 but op-reth forces it to 0 (ethereum-optimism/protocol-team#457). Forkchoice updates whose safe or finalized hash is not on the head's chain are rejected (reth#27247, reth#27429). --txpool.minimum-priority-fee now also applies to local and conditional transactions (reth#26861). eth_getLogs returns error 4444 for a range whose receipts are missing and -32602 for a fromBlock beyond the head (reth#27002, reth#26891). Peers negotiate eth/71 and the transaction fetcher was rewritten (reth#26924). Block access list flags and RPC methods are rejected. The deferred persistence handoff from the fork fix reth#26708 is replaced by upstream's overlay snapshots, and its regression test rust/op-reth/crates/node/tests/it/payload_handoff.rs was deleted.
• Existing coverage. CI on the bump head is green for memory-all-opn-op-reth, memory-all-kona-op-reth, rust-e2e-restart, op-reth-e2e-sysgo-tests and op-reth-integration-tests. Acceptance tests restart nodes only gracefully (nothing sends SIGKILL), several EL-restart and EL-sync tests are SkipOnOpReth, no test crosses a persistence boundary on purpose, and op-acceptance-tests/tests/flashblocks has no tests.
• Devstack seams. sysgo.OpRethWithBinary selects a per-node binary (resolved through RUST_BINARY_PATH_<NAME>), OpRethWithExtraArgs adds flags, and a follower gets its options through MixedSingleChainNodeSpec.OpRethOpts (presets.WithOpRethOption reaches only sequencer-capable ELs). Missing today: a hard kill (SubProcess.Stop(false) only waits; add Kill()), an L2 CL sync-mode option in the mixed runtime, and a hook to assert on op-reth log lines. op-reth: acceptance test for proofs history across an unclean shutdown #23380 adds the kill seam and op-reth: add a windowed proof oracle to the devstack DSL #23379 the proof oracle; depend on them rather than duplicating.
• Conventions. docs/ai/writing-acceptance-tests.md, docs/ai/acceptance-tests.md, docs/ai/rust-dev.md, and docs/ai/reth-update-review.md (precondition 6: a per-account storage root is consensus on OP, so test it with persistence engaged). Scoped commits; run rust-code-reviewer or go-code-reviewer before opening the PR.
• Target branch. Write against develop where the behaviour exists there. A test that only makes sense on the bump is written against rust!: update reth to v2.6.0 #23046's branch and says so in the PR. A test that fails for a tracked reason is kept and skipped naming the issue.

Parameter matrix (applies to every sub-issue)

Persistence-related tests run over reth threshold {7 (develop), 50 (bump)} via --engine.persistence-threshold, masking {0, 30 when reachable (ethereum-optimism/protocol-team#457)}, and where the CL matters {op-node, kona-node} (CI runs both). Go tests use table-driven subtests with the configuration in the subtest name; Rust tests use rstest tables, one case per line. Every skipped cell says why in the skip message.