op-devstack: add a windowed proof oracle to the L2 EL DSL

[einar-oplabs]

Why

Every proofs-store corruption so far (wipe handling, masked-trie overlays, the withdrawalsRoot bug, the crash gap) shows up as a wrong eth_getProof for some historical block. This adds one DSL sweep that catches all of them without knowing the cause, so the proofs-history scenario tests (#23380, #23383, #23389, #23390, #23395, #23396, #23398) can share a single oracle.

What

op-devstack/dsl/l2_el_proofs.go

• L2ELNode.VerifyHistoricalProofsMatch(reference, opts...): reads the node's proofs window from debug_proofsSyncStatus, waits (bounded) for the store to reach the sweep's last block, then for every sampled block fetches eth_getProof by block hash for L2ToL1MessagePasser plus the configured accounts from the node under test and from reference, Merkle-verifies both against the header's state root, requires them equal after normalization, and requires the message passer's storage root to equal the header's withdrawalsRoot (post-Isthmus, i.e. whenever the header value is not the empty withdrawals hash). Headers are fetched from both nodes and must agree on the hash. The window is re-read at every block: a block the pruner removes during the sweep is skipped when the caller did not pin FirstBlock, and fails the sweep otherwise. The reference must answer debug_proofsSyncStatus with method-not-found, which only a node without a proofs-history store does.
• HistoricalProofsOpts: Accounts []ProofAccount (address + storage keys), CatchUpTimeout (default 30s), FirstBlock/LastBlock (zero = window edge at sweep start), MaxSampledBlocks (zero = every block; edges always included).
• VerifyProofsMatchAt(reference, block, accounts...): single-block form (listed accounts only, no independence check); ProofsSyncStatus() and WaitForProofsStoreBlock(target, timeout) expose the sync status RPC.
• NormalizeProofResponse and VerifyAccountProof moved here from rust/op-reth/tests/proofs/utils/proof.go. The normalizer additionally turns an empty storage proof list into nil so geth and reth responses compare equal; the verifier additionally rejects a nonzero storage value claimed under an empty storage root. utils/proof.go and utils.WaitForProofsStoreBlock are deleted; proofs/core, proofs/prune and proofs/reorg call the DSL methods instead and drop the waits VerifyProofsMatchAt performs itself.

op-devstack/sysgo

• OpRethConfig.ProofsHistoryWindow + OpRethWithProofsHistoryWindow(blocks): the runtime passes --proofs-history.window itself and op-reth rejects a repeated flag, so the window is a config field (proofsHistoryNodeArgs emits it once; default stays 10000). op-reth: acceptance test for the production proofs init path with backfill #23389 and op-reth: tests for the proofs RPC window edges and old-block witness requests #23390 reuse it.
• OpRethWithEthProofWindow(blocks): reth's --rpc.eth-proof-window defaults to 0, so a node without proofs history rejects every historical eth_getProof with ExceedsMaxProofWindow. A reference node needs this.

op-acceptance-tests/tests/proofs_history_match

TestProofsHistoryFollowerServesMatchingHistoricalProofs: a sequencer started with OpRethWithoutProofsHistory() + OpRethWithEthProofWindow is the independent reference; a follower runs proofs history under the matrix cell's flags. Four rounds of: two EOAs transfer back and forth, one initiates a withdrawal (so the message passer's storage root changes inside the swept range), and one calls a contract that SSTOREs NUMBER and TIMESTAMP. Before the last round the test waits for the follower to pass persistence-threshold + 2 * memory-block-buffer-target blocks, so reth has persisted at least once and the sweep spans blocks on both sides of that. The follower is then swept from the window's first block to the last workload block for the message passer, both EOAs and the contract's two slots.

Matrix (table-driven subtests): persistence threshold {7, 50} x state masking {0, 30} x proofs path {index = verification-interval=0, execute = verification-interval=1}. The four masking_30 cells skip naming ethereum-optimism/protocol-team#457 (the flag does not exist on develop's reth 2.5.2 build and #23046 rejects a nonzero value).

Unit tests: VerifyAccountProof against a synthetic state trie (valid account with storage, without storage, absent account, zero slot under an empty root; tampered balance, tampered storage value, nonzero slot under an empty root, absent account claiming a balance, wrong root), NormalizeProofResponse (five cases), ProofsSyncStatus.serves (five), sampleBlockNumbers (seven), proofsHistoryNodeArgs and the two new options.

Local verification

go test -count=1 ./op-devstack/dsl/ -run 'TestVerifyAccountProof|TestNormalizeProofResponse|TestSampleBlockNumbers|TestProofsSyncStatusServes'   # ok
go test -count=1 ./op-devstack/sysgo/ -run 'TestProofsHistoryNodeArgs|TestOpRethWith'                                                         # ok

DEVSTACK_ORCHESTRATOR=sysgo RUST_BINARY_PATH_OP_RETH=.../target/debug/op-reth \
  go test -count=1 -timeout 40m -parallel 2 -v -run TestProofsHistoryFollowerServesMatchingHistoricalProofs \
  ./op-acceptance-tests/tests/proofs_history_match/
--- PASS: threshold_7-masking_0-index (58.61s)      sweep: 26 blocks x 4 accounts
--- PASS: threshold_7-masking_0-execute (56.88s)    sweep: 25 blocks x 4 accounts
--- PASS: threshold_50-masking_0-index (136.96s)    sweep: 66 blocks x 4 accounts
--- PASS: threshold_50-masking_0-execute (154.70s)  sweep: 65 blocks x 4 accounts
--- SKIP: threshold_{7,50}-masking_30-{index,execute}   (protocol-team#457)

An earlier run without LastBlock pinned showed the catch-up wait firing (Proofs store behind target ... window [0, 20]) while the sweep's last block was still buffered in the ExEx.

Proofs e2e suite after the migration (rust/op-reth/tests/proofs/core, both nodes op-reth, debug binary):

--- PASS: TestL2MultipleTransactionsInDifferentBlocks (37.23s)
--- PASS: TestL2MultipleTransactionsInSingleBlock (27.24s)
--- PASS: TestExecutePayloadSuccess (17.41s)
--- PASS: TestExecutePayloadWithInvalidParentHash (17.63s)
--- PASS: TestDebugExecutionWitness (26.82s)
--- PASS: TestResyncing (43.42s)
--- PASS: TestStorageProofUsingSimpleStorageContract (46.79s)
--- PASS: TestStorageProofUsingMultiStorageContract (67.78s)
--- PASS: TestTokenVaultStorageProofs (37.37s)
ok  	github.com/ethereum-optimism/optimism/rust/op-reth/tests/proofs/core	322.780s
EXIT=0

Not run: the kona-node CL variant (DEVSTACK_L2CL_KIND=kona-node; no kona-node binary built locally). The prune and reorg packages compile but stay dormant as before.

Corruption check (not committed)

A copy of the test started the follower with sysgo.OpRethWithProofsHistoryWindow(20), ran three times the workload so the pruner moved earliest to 25, and swept with opts.FirstBlock = 1:

Sweeping historical proofs  name=proofs-follower reference=sequencer window="[25, 45]" first=1 last=45 blocks=45 accounts=4
Error:      Should be true
Messages:   proofs store on proofs-follower does not serve block 1: window [25, 45]
--- FAIL: TestCorruptionCheckSmallerWindowBites (101.42s)

The sweep fails before any account is fetched, because the store no longer serves the block at all; a wrong proof for a served block fails one level deeper with block N account 0x..: eth_getProof from <node> does not verify against state root ... / ... differs from reference <node>, which the unit tests exercise with tampered proofs.

Findings

• op-reth's eth_getProof rejects a null storage-key list (Invalid params: invalid type: null, expected a sequence) where op-geth accepts it. The old helpers always passed []common.Hash{}; the DSL now sends an empty list for a nil StorageKeys.
• While that bug made every eth_getProof fail fast, three of four cells ended their 30s retry window with dial tcp 127.0.0.1:<proxy-port>: i/o timeout against the follower's user-RPC proxy, the symptom safeheaddb_elsync already skips on op-reth. It did not recur once requests stopped failing; noted in case it shows up again.
• Deviation from the issue text: the issue assumes the devstack sequencer always runs proofs history and asks for a separate reference node. The test instead turns proofs history off on the sequencer (MixedSingleChainNodeSpec.OpRethOpts), which is the production shape and saves a third node. The verification criterion expects the smaller-window failure to name "the block and account"; the oracle fails earlier, at the window check, naming the block, the node and the window (see above).

Seams for the sibling PRs

• L2ELNode.VerifyHistoricalProofsMatch / VerifyProofsMatchAt / WaitForProofsStoreBlock / ProofsSyncStatus.
• sysgo.OpRethWithProofsHistoryWindow, sysgo.OpRethWithEthProofWindow (a reference node needs the latter together with OpRethWithoutProofsHistory).

Closes #23379

🤖 Generated with Claude Code