feat(protocols): Add code to parse and process ContractData entires for Blend processors - #657
feat(protocols): Add code to parse and process ContractData entires for Blend processors#657aditya1702 wants to merge 14 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4233173ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Introduces a new optional “framework seam” for protocol processors to consume ContractData ledger-entry changes, gated behind a RequiresContractData() capability so event-only protocols avoid the heavier extraction path. This supports upcoming Blend Capital v2 lending ingestion by making ContractData deltas available to processors without forcing extraction for all protocols.
Changes:
- Extends
ProtocolProcessorwithRequiresContractData() booland extendsProtocolProcessorInputwithContractDataChanges map[string][]ingest.Change. - Adds
indexer.ExtractContractDataChangesForLedgerand wires it into both migration (processAllProtocols) and live ingestion (lazy/memoized per ledger). - Adds/updates mocks and tests to validate the capability gate and fixture-based correctness of the extractor.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/services/sep41/processor.go | Implements RequiresContractData() as false for SEP-41 (event-only). |
| internal/services/protocol_processor.go | Adds the capability gate method and ContractDataChanges to processor input. |
| internal/services/protocol_migrate.go | Extracts ContractData changes (conditionally) and passes them into processors during migration. |
| internal/services/protocol_migrate_test.go | Updates test processors/mocks and adds migration tests for ContractDataChanges gating. |
| internal/services/processor_registry_test.go | Adds a mock-based test covering the new interface method. |
| internal/services/mocks.go | Extends ProtocolProcessorMock with RequiresContractData(). |
| internal/services/ingest_test.go | Adds live-ingestion tests for nil/non-nil ContractDataChanges based on the gate. |
| internal/services/ingest_live.go | Lazily extracts and memoizes ContractData changes per ledger when required by a CAS-winning processor. |
| internal/indexer/indexer.go | Adds ExtractContractDataChangesForLedger grouped by owning contract C-address. |
| internal/indexer/indexer_test.go | Adds fixture-corpus test validating extraction correctness and grouping invariants. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
325123d to
a5acea9
Compare
aditya1702
changed the title
|
Race at the migration/live frontier: a contract classified in the ledger being folded can permanently lose its deploy-ledger ContractData This is latent today (the only in-tree processor is event-only — The asymmetry Both producers of protocol state need contract membership, because ContractData extraction is footprint-gated:
The failure timeline Say a Blend-style processor tracks pools P1, P2, and migration has caught up to the frontier (cursor = 1999, window size 1 via
End state: P3 is a tracked contract, its post-2001 changes flow normally, but its deploy-ledger entries exist on chain and nowhere in the DB. Never-rewritten keys (immutable pool config) never heal; keys a later ledger rewrites heal in current-state but stay missing from history. No error is raised anywhere. The race is confined to the frontier — below the tip, a contract deployed at tip T has no footprint in older windows, and membership refreshes pick it up long before migration reaches T. But "migration catches up to tip while a new pool deploys via the permissionless factory" is exactly the window this handoff design exists for. Possible fixes Refreshing harder can't close this — committed-row reads can't see an uncommitted classification. The options change who owns the deploy ledger:
Related: |
ae6f1d4 to
8f7b101
Compare
|
Re: the migration/live frontier race (#657 (comment)) — implemented option 1 (live-side repair) in ba7ebfb. On a lost protocol-cursor swap where this ledger classified new contracts for the protocol, live now checks the lost cursor's committed value in-transaction (new Two deltas from the write-up:
Accepted residual (documented on Covered by CAS-gating cases M1–M4: frontier repair, behind-tip non-repair, partial-loss scoped repair, already-committed exclusion; mutation-verified. |
|
Follow-up on the frontier race — one more timing crack closed, in 0f0692a. The engine refreshed membership immediately after each window commit. At the contested ledger L that refresh fires milliseconds after the engine's commit — but live's losing transaction (which carries the new contract's classification) commits after that, since its CAS was lock-blocked behind the engine and it still has the repair and the rest of its persist to run. So the engine could stage L+1 with a snapshot still missing the contract, and if it won L+1 too, live's repair wouldn't fire there (the contract is no longer newly-classified at L+1). One ledger of the contract's events could slip through — and the additive fold columns (cost basis, lifetime claimed) never heal from a missed ledger, unlike the snapshot columns. The refresh now runs at window start, after The mid-window residual from the previous comment is unchanged — a window staged over many ledgers still uses one snapshot; that one needs engine-side replay. |
… behind the capability gate
…d the capability gate
…sors Live ingestion derived ProtocolContracts from this ledger's event emitters only. A ContractData-requiring processor needs the protocol's complete committed membership: entries can change on contracts that emitted no event this ledger, and event decoding disambiguates shared symbols against the full tracked set (e.g. backstop vs pool withdraw).
…data extract metric phase
A processor enriching protocol_contracts (contract names decoded from instance storage) inserts rows FK-filtered against protocol_wasms; a contract deployed in the same ledger as its wasm upload was silently dropped because the wasm rows persisted after the processor block.
…ta processors The migration engine loaded each tracker's classified-contract membership once before the unbounded ledger loop. A contract classified while the run is in flight (live ingestion's validator classifies newly deployed contracts concurrently) never reached membership-driven processors: for BLEND, a pool deployed mid-run lost its history rows permanently and mis-resolved the pool-vs-backstop withdraw disambiguation for the rest of the run. Trackers whose processor requires ContractData now re-read membership via GetByProtocolID after every committed window (both in-loop and at-tip flushes), mirroring the live path's per-ledger full-membership resolution. Event-only processors keep the run-start snapshot.
…extraction Live ingestion materialized every ledger's transactions twice: once in the main indexing pass and again inside ExtractContractDataChangesForLedger, whose reader constructor re-hashes every transaction envelope. ProcessLedger now returns the transactions it already built, and the live path hands them to a new ExtractContractDataChangesFromTransactions, so a ledger is read exactly once. The ledger-based extractor remains as a thin wrapper for protocol-migrate, which has no prior transaction pass to share.
protocol-migrate spent ~7ms/ledger in extract_contract_data — a LedgerTransactionReader build (SHA-256 of every envelope) plus a full GetChanges walk of every transaction — on every ledger of the range, though almost none touch a tracked contract. Soroban guarantees writes ⊆ the declared read-write footprint (host storage is footprint-seeded and writes outside it trap; RestoreFootprint restores exactly the read-write keys; protocol-23 auto-restores are indices into it), so a skim of the already-decoded envelopes' footprints decides ledger relevance exactly. Ledgers whose footprints touch no tracked contract skip extraction outright; the rare hit extracts via transaction meta directly — GetChanges reads only meta/result/ledger-version, so no reader is built even then. The fixture-corpus test now pins both properties on real ledgers: meta-based output equals the reader-based reference, and every changed contract, tracked alone, triggers the gate. Live ingestion keeps the ungated slice-based path over its already-materialized transactions.
GetChanges attaches the transaction to every Change it returns, so the footprint-gated extraction path handed processors transactions with a zero Hash while the reader-fed path populates it. Mirror the reader: resultPair.TransactionHash is the exact value it reads. Also scope the equivalence-test comment to the change fields the fixture actually compares — the attached transaction is projected away.
It asserted a testify mock returns what it was programmed to return; the gate is covered behaviorally by protocol_migrate_test.go (populated vs left nil) and ingest_test.go cases H/I/J, and the compile-time interface assertion already exists in mocks.go.
A concurrent protocol-migrate engine snapshots membership from committed protocol_contracts rows only, so when it wins a ledger's cursor swap it has folded that ledger without contracts whose classification commits with live's in-flight transaction — their deploy-ledger state (constructor ContractData writes, first events) was extracted by nobody, and cursor-passed ledgers are never replayed. Never-rewritten keys stayed missing forever. On a lost swap where this ledger classified new contracts for the protocol and the lost cursor's committed value is at or past this ledger (a lost CAS blocked on the winner's row lock, so that value is reliably visible in-transaction via the new IngestStore.GetInTx), live now re-stages scoped to exactly the gap contracts and persists the lost halves in the same transaction, without moving any cursor. The CAS-winning path is extracted into stageAndPersistProtocolLedger alongside the new repairClassificationGap. Covered by CAS-gating cases M1-M4: frontier repair, behind-tip non-repair, partial-loss scoped repair, and already-committed exclusion.
…r is fetched The engine refreshed a tracker's classified-contract membership right after each window commit — milliseconds before a concurrent live transaction for that same contested ledger finishes committing a new contract's classification, since live's lost CAS was blocked behind the engine's row lock and still has work to do after unblocking. The engine then staged the next window with a snapshot missing that contract, skipping its events and entries for one more ledger; additive fold columns (cost basis, lifetime claimed totals) never heal from a missed ledger, so those deltas were permanently lost whenever the engine also won that next ledger. The refresh now runs at window start, after GetLedger returns for the window's first ledger: the fetch blocks until that ledger has closed, which is a full ledger interval after any concurrent transaction for the previous one committed. Same cadence — once per window per requiring tracker — with the read taken at the latest useful moment. The refresh test now pins the ordering: the first folded ledger must already carry membership committed after the run-start snapshot (mutation-verified — removing the window-start refresh fails it).
0f0692a to
d9aac77
Compare
|
Re: the classification-gap repair (#657 (comment)) — I filed #680 for the residual in the The
If live lags the tip by k ledgers:
Ledgers L+1 to L+k are folded by nobody. The gap equals live's lag. This is deterministic, not timing dependent. The gap does not close by itself. The engine hands off only when its own CAS fails ( Impact:
This is not a regression. The repair improves the previous behavior. #680 has the proposed fix and the scope analysis. |
|
|
||
| if repairHistory { | ||
| persistStart := time.Now() | ||
| if persistErr := processor.PersistHistory(ctx, dbTx); persistErr != nil { |
There was a problem hiding this comment.
repairClassificationGap can abort live ingestion with a primary key violation.
The chain:
state_changeshasPRIMARY KEY (ledger_created_at, to_id, operation_id, state_change_id)(2025-06-10.4-statechanges.sql:49).- The repair writes history for a ledger whose cursor it lost. The winner already wrote history for that same ledger.
- Both producers derive
to_idandoperation_idfromtoid.New(ledgerSeq, txIdx, opIdx+1)(sep41/processor.go:127), and both use the sameledger_created_at. AssignStateChangeOrdinalsrestarts its counter at 1 for each(ToID, OperationID)pair and adds the protocol base (types.go:822). The counter is local to one staged slice.- The first row each producer stages for a shared operation gets
base+1. BatchCopyusespgxTx.CopyFromwith noON CONFLICT(statechanges.go:158).- SQLSTATE 23505.
isPermanentPersistErrortreats class 23 as permanent (ingest_live.go:1006), so there is no retry. Live ingestion stops on that ledger.
Precondition: one operation emits events from both a gap contract and a contract in the winner's membership. A Soroban transaction normally carries one InvokeHostFunction operation, so all its contract events share one opID. A factory that deploys a token with a minting constructor and moves an existing token in the same transaction meets this.
PersistCurrentState is safe. Balance and allowance keys include the contract ID, and processEvent filters on the emitting contract (sep41/processor.go:153), so repair rows and winner rows never share a key.
Root cause: the CAS guaranteed one producer per (ledger, operation). Ordinal assignment depends on that guarantee. The repair writes outside it. The doc comment's "additive across disjoint membership sets" covers row content, not the ID namespace.
Fix — the pattern already exists in types.go. The indexer splits its 2^40 namespace at bit 28 into 4096 sub-streams. Protocol emitters are documented as single-stream, with a note that a future multi-stream emitter subdivides its own namespace the same way. The repair is that second stream. Give it a distinct sub-base inside the protocol's namespace. Repair rows then cannot collide with winner rows. This is deterministic, needs no extra query, and needs no coordination between producers.
Smaller alternative: drop the history half of the repair and keep current state only. That removes the collision. It also removes the deploy-ledger history rows, which is the main reason the repair exists.
This finding is independent of #680. The gate proposed there removes the cursor > L case. The cursor == L case still runs the repair, so the collision survives that fix.
| } | ||
| rawIDs = append(rawIDs, raw) | ||
| } | ||
| committedRows, err := m.models.ProtocolContracts.BatchGetByContractIDs(ctx, rawIDs) |
There was a problem hiding this comment.
GetByProtocolID was moved to db.Querier in dd2cee5, so this call looks like the one that was missed. Making the same change here would silently disable the repair.
The reason is the write ordering documented at :123-126: a processor's Persist* inserts name-enriched protocol_contracts rows before the generic insert at :238. In the partial-loss case:
stageAndPersistProtocolLedgerpersists the won half for protocol P.- The gap contract C is in that processor's membership, through the
getEffectiveProtocolContractsoverlay. - An enriching processor writes C's row into
dbTx. - On
dbTx, this query returns C.alreadyTrackedthen drops C,gapis empty, and the repair returns at:439.
The pool connection does not see the uncommitted row, so C stays in gap and the repair runs. No in-tree processor enriches yet, so both paths agree today. Blend is the expected first one.
Two suggestions:
- State in the comment that the read must not see this transaction's own writes, and that
dbTxwould suppress the repair. - Add an M5 case: partial loss where the won half's processor inserts a
protocol_contractsrow for the gap contract, asserting the repair still persists. That turns the comment into a guard.
JiahuiWho's pool-saturation point still stands as written, and is unchanged by this: single-threaded today, and this read is behind the len(candidates) == 0 return at :409.
| if processErr := processor.ProcessLedger(ctx, input); processErr != nil { | ||
| return fmt.Errorf("processing classification-gap repair at ledger %d for protocol %s: %w", ledgerSeq, protocolID, processErr) | ||
| } | ||
| m.appMetrics.Ingestion.ProtocolStateProcessingDuration.WithLabelValues(protocolID, "process_ledger").Observe(time.Since(start).Seconds()) |
There was a problem hiding this comment.
The repair reuses the process_ledger, persist_history, and persist_current_state phase labels.
In the partial-loss case (M3), stageAndPersistProtocolLedger observes process_ledger at :337 and the repair observes it again at :492. One protocol emits two samples for one ledger.
This is the same problem Copilot raised about extract. The fix in 1849ed9 added a distinct extract_contract_data label, described as "keeping extract at one observation per ledger". The repair reintroduces the pattern on three labels.
There is a second effect. When both halves are lost (M1) only the repair runs, and its samples are indistinguishable from normal staging. Repair activity cannot be counted, graphed, or alerted on. Given #680, repair frequency is a useful signal.
Suggested fix, matching the 1849ed9 precedent:
- Observe the repair under distinct phases:
repair_process_ledger,repair_persist_history,repair_persist_current_state. - Update the Help text at
metrics/ingestion.go:138. It enumerates the valid phase values.
Separately, the Observe calls sit after the error returns here (:492, :498, :506). In stageAndPersistProtocolLedger they sit before (:337, :345, :353), so a failed call is still recorded. A failed repair records no duration. Please match the existing path.
Framework:
ContractDataChanges+ capability gateFirst of 5 stacked PRs adding Blend Capital v2 lending support.
Adds
RequiresContractData() booltoProtocolProcessorand aContractDataChanges map[string][]ingest.Changefield onProtocolProcessorInput, populated by the new footprint-gated extraction ininternal/indexer:Post == nil; ledger-level archival evictions are not surfaced. Contract-id encode errors fail fast instead of silently dropping changes. Synthesized transactions carry the reader-identicalHash(TxProcessing[i].Result.TransactionHash), sinceGetChangesattaches the transaction to every change it returns.LedgerTransactionReader— SHA-256 of every envelope — on every ledger of the range); a hit extracts directly from transaction meta. A fixture corpus over real pubnet ledgers (169 grouped changes across 5 ledgers, one pre-CAP-46-11 classic-account-owned entry documented as skipped) pins meta-based output ≡ the reader-based reference on the change fields, and that every changed contract, tracked alone, triggers the gate.processAllProtocols) extracts once per ledger, only when a tracker that will fold the ledger has a processor requiring it; all trackers share the map. Contract membership refreshes once per window, read at window start after the window's first ledger is fetched — the fetch blocks until that ledger closes, so the read lands after any concurrent live transaction for the previous ledger (and the classification it carries) has committed; a contract deployed mid-run starts folding from the next window. Timed into a distinctextract_contract_datametric phase.RequiresContractData()processors receive the FULL committed protocol membership (GetByProtocolID), not just this ledger's event emitters: entries can change without events, and event decoding disambiguates against the full tracked set.protocol_contractsrows only, so when it wins a ledger's cursor CAS it has folded that ledger without contracts whose classification commits inside live's concurrent transaction — their deploy-ledger state (constructor ContractData writes, first events) would be persisted by nobody, permanently for never-rewritten keys. On a lost swap where live classified new contracts for the protocol and the lost cursor's committed value is at or past the ledger (a lost CAS blocked on the winner's row lock, so that value is reliably visible via the newIngestStore.GetInTx), live re-stages scoped to exactly the gap contracts and persists the lost halves in the same transaction, without moving any cursor. Covered by CAS-gating cases M1–M4 (frontier repair, behind-tip non-repair, partial-loss scoped repair, already-committed exclusion). Accepted residual: an engine mid-window across the deploy ledger concurrent with live's classification commit (requires live to lag the engine by a full window; current-state heals on the contract's next write, that window's history rows stay missing) — documented onrepairClassificationGap.false— event-only protocols are unaffected.Deviations register (reviewer sign-off):
processor,ProtocolProcessorMock,testRecordingProcessor, plustestProtocolProcessor(ingest_test.go, discovered viago vet)🤖 Generated with Claude Code