feat(svm): Solana (SVM) indexing support - #1287
Merged
Merged
Conversation
Replaces the runtime `@envio-dev/hypersync-client` npm dependency with
in-tree NAPI bindings that wrap the upstream `hypersync-client` Rust
crate, exposed off the existing `envio.node` cdylib. Removes one
3rd-party native dep (plus its five per-platform binaries) from
hyperindex installs.
Surface kept minimal to what HyperIndex actually calls:
`HypersyncClient.{new, newWithAgent, get, getEvents}`,
`Decoder.{fromSignatures, decodeEvents}`, and module-level
`setLogLevel`. Decoder checksum behaviour simplified to a construction
flag (`Decoder.fromSignatures(sigs, ~checksumAddresses=...)`); the
former runtime enable/disable methods are gone.
Bumps `schemars` to 1.2 (required transitively by `hypersync-client`)
and updates `human_config.rs` for its renamed trait method.
Adds a Solana counterpart to packages/cli/src/hypersync_source/, wrapping the
upstream hypersync-client-solana 0.0.2-rc.1 crate and exposing a
`HypersyncSolanaClient` napi class off the existing envio.node cdylib.
Surface kept minimal: `new`, `getHeight`, `get` (which wraps the upstream
`Client::collect` for paginated single-call queries). Query and response
shapes mirror the upstream net-types, with JS-friendly numerics (i64 for
slots and indices, `0x`-prefixed hex strings for instruction data and the
d1/d2/d4/d8 discriminator prefixes, base58 strings for pubkeys).
- packages/cli/Cargo.toml: add hypersync-client-solana, hypersync-solana-net-types.
- packages/cli/src/hypersync_source_svm/: new module
- config.rs: napi ClientConfig with url, api_token, timeout, retries.
- query.rs: napi SolanaQuery + InstructionSelection / TransactionSelection /
LogSelection / FieldSelection, with TryFrom into the upstream net-types.
Field-selection enums coerced from strings.
- types.rs: flat napi response shapes (Block/Transaction/Instruction/Log)
converted from upstream simple_types.
- mod.rs: HypersyncSolanaClient + #[ignore]-gated live smoke test
(verified locally: 390 Metaplex Token Metadata instructions decoded
from a 10k-slot window against solana.hypersync.xyz).
Stacks on PR #1212 (claude/rust-client-hyperindex-Nr6pt). No changes to the
EVM hypersync_source module; only one trivial fmt drift restored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a ReScript wrapper around the `HypersyncSolanaClient` napi class that PR #1214 registers on the envio.node addon. Mirrors the shape of HyperSyncClient.res: - `cfg` record with url + optional auth/timeout/retry knobs. - `QueryTypes`: field enums (block/transaction/instruction/log) plus the selection records (`instructionSelection` carries `programId`, `d1`..`d8` hex prefixes, `a0`..`a9` account filters, `isInner`, `includeTransaction`, `includeLogs`; `transactionSelection` and `logSelection` analogously). - `ResponseTypes`: typed block/transaction/instruction/log records, with instruction `data` and discriminator prefixes as `0x`-prefixed hex and pubkeys as base58 strings. - `make` constructor + a `%raw` wrapper for the JS `new` operator (the napi class is grabbed dynamically off `Core.getAddon()`, so `@new` can't bind to a name). Wires the new `hypersyncSolanaClient` constructor onto the addon record in Core.res, alongside `hypersyncClient` and `decoder`. Adds a `describe_skip`-gated live test at `scenarios/test_codegen/test/HyperSyncSolanaClient_test.res` that hits `solana.hypersync.xyz`, filters on the Metaplex Token Metadata program for the last ~10k slots, and verifies the returned instructions decode through the napi -> ReScript path. Verified locally end-to-end (vitest run passed, ~5s). Out of scope here: - HyperSyncSolanaSource.res (Source.t implementation): deferred until the config schema (Stage 3) and handler-dispatch model (Stage 4) exist, since bridging Solana instructions into Internal.item (today: Event | Block) is the Stage 4 work. Stacks on PR #1214 (claude/solana-hypersync-napi-binding). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the SVM YAML schema (previously RPC-only with no contracts) to support
declaring Solana programs and the instructions to index on each. Mirrors the
EVM/Fuel "contracts -> events" shape, adapted to Solana.
human_config.rs (svm module):
- HypersyncConfig { url } — optional per-chain HyperSync endpoint, mirrors EVM.
- Program { name, program_id, handler?, instructions } — name drives codegen,
program_id is the base58 pubkey.
- Instruction { name, discriminator?, is_inner?, account_filters?,
include_transaction?, include_logs? } — discriminator is hex (1/2/4/8 bytes
including 8-byte Anchor), account_filters pin positional accounts (0..=9).
- AccountFilter { position, values }.
- Chain gains optional hypersync_config + programs.
system_config.rs:
- DataSource::Svm gains hypersync_endpoint_url: Option<String>; populated from
the parsed YAML. Validation runs before chain construction.
public_config.rs:
- SVM branch emits the HyperSync URL alongside the RPC URL so the runtime JSON
carries both. (Config.res still only reads `rpc` today; consuming `hypersync`
is wired up in Stage 4 along with the handler-dispatch model.)
validation.rs:
- is_valid_solana_pubkey: base58 alphabet + 32..=44 char length sanity check.
- validate_svm_discriminator: hex with optional 0x prefix, 1/2/4/8 bytes.
- validate_deserialized_svm_config_yaml: walks programs + instructions,
enforces unique program names + unique instruction names per program,
position in 0..=9, valid base58 for account-filter values.
svm.schema.json: regenerated via `cargo run --example script -- script
print-config-json-schema svm`. The existing test_svm_config_schema test
verifies the regenerated schema matches schemars' output.
Tests:
- 7 new tests in `validation::tests::svm` cover the validation surface
(valid + invalid pubkey, discriminator length + chars, duplicate program
names across chains, duplicate instruction names within a program,
account_filter position bounds, bad program_id).
- 2 new tests in `human_config::tests::svm_yaml` cover the round-trip
(Metaplex Token Metadata example yaml fully deserializes; unknown fields
rejected).
- All 153 config_parsing tests pass; no regressions in EVM or Fuel.
Out of scope (Stage 4):
- Translating programs into runtime Contract/Event structures for codegen.
- Handler API + dispatch (`<Program>.<Instruction>.handler(...)`).
- Wiring the new HyperSync URL into ChainFetcher source construction.
Stacks on PR #1215 (claude/solana-hypersync-rescript-source).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end plumbing from YAML programs/instructions through to a typed
`Internal.svmInstructionEventConfig` ReScript runtime config. Handlers
compile but the actual dispatch (HyperSyncSolanaSource, ChainFetcher
wiring, EventRouter) lands in C2.
Rust side:
- `system_config::EventKind::Svm(SvmEventKind)` carries discriminator,
discriminator_byte_len, include_transaction, include_logs, account_filters,
is_inner. New `SvmAccountFilter`.
- `Abi::Svm` unit variant on the per-contract abi enum; Solana programs ship
no ABI artifact today (Borsh schema lands per the Stage 7 roadmap).
- `system_config.rs` HumanConfig::Svm arm walks each program/instruction,
producing a `Contract` with `Abi::Svm` and one `Event{kind: EventKind::Svm,
sighash: discriminator}` per instruction. ChainContract.addresses[0] holds
the base58 program_id.
- `public_config.rs` extends `ContractEventItem` with optional `svm`
descriptor + extends SvmConfig to carry `programs` alongside `chains`. The
per-event JSON now ships the SVM flags the runtime needs.
- `codegen_templates.rs`: new `EventTemplate::from_svm_instruction_event`
emits a minimal per-instruction ReScript module (`event` /
`paramsConstructor` / `onEventWhere` aliases) — enough surface for the
GADT to type-check. The rich `indexer.onInstruction` GADT registration is
C2 work.
- `contract_import_templates.rs`: SVM arm yields empty params (the
contract-import flow is EVM/Fuel-only).
ReScript side:
- New `SvmTypes.res` with a thin `SvmTypes.Pubkey.t` newtype (per the
Q4 review answer; treats Solana pubkeys distinctly from EVM `Address.t`).
- `Internal.svmInstructionEventConfig` mirrors `evm`/`fuelEventConfig`,
carrying programId + discriminator + flags so the future router can
dispatch by `(programId, discriminator)`.
- `Envio.res`: public `svmInstruction`, `svmTransaction`, `svmLog`,
`svmInstructionEvent`, `svmOnInstructionArgs<'context>`. Mirrors EVM's
`{event, context}` shape — the per-instruction `event` payload carries
`instruction`, `transaction?`, `logs?`, `slot`, `blockTime?`.
- `EventConfigBuilder.buildSvmInstructionEventConfig` is the runtime
constructor; `Config.res` Svm arm calls it from `buildContractEvents`.
- `HandlerLoader.applyRegistrations` Svm arm replaces the previous throw
with a Fuel-style pass-through (registration plumbing now ready for C2's
dispatch wiring).
Note on user-facing API: the locked roadmap originally described a
`Contract.Event.handler(...)` style, but EVM/Fuel actually use
`indexer.onEvent({contract, event}, handler)`. Mirroring EVM more honestly,
the C2 surface will be `indexer.onInstruction({program, instruction}, handler)`.
The generated per-instruction ReScript modules expose the GADT-friendly
type aliases now so adding that method in C2 is additive.
Tests:
- `cargo test -p envio --lib` — 154 passing (1 new SVM translation test
exercises the Metaplex YAML fixture end-to-end through
parse → validate → translate, asserting the Contract / Event /
Chain shape).
- `pnpm rescript` — 113 modules compile clean (added SvmTypes.res and
extended Envio.res / Internal.res / EventConfigBuilder.res /
HandlerLoader.res / Config.res).
- New fixture: `packages/cli/test/configs/svm-metaplex-config.yaml`.
Stacks on PR #1216 (claude/solana-hypersync-yaml-config).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makes the SVM ecosystem live: programs/instructions declared in YAML now
flow all the way through to the user-facing `indexer.onInstruction(...)`
handler.
Source layer (HyperSyncSolanaSource.res):
- Source.t implementation. Builds one `InstructionSelection` per
`(programId, discriminator)` declared in the config: the matching dN
field (d1/d2/d4/d8) carries the discriminator; a0..a5 carry positional
account filters (matching the C1 napi cap); `isInner`,
`includeTransaction`, `includeLogs` flow through.
- Per-(slot, tx_idx) transaction lookup and per-(slot, tx_idx,
instruction_address) log grouping so each handler sees only the logs
scoped to its instruction (Q2 answer — per-instruction grouping).
- Probes EventRouter longest-discriminator-prefix first via the per-program
byte-length ordering precomputed at router-build (Q1 answer).
- Synthesized logIndex `tx_idx * 65536 + depth-weighted addrSum` keeps
FetchState ordering deterministic without touching its compare logic.
- No-op reorg guard for C2 (Q3 deferred to C3 since it needs an extra
`queryBlockHash` route on the napi client).
Routing helpers (EventRouter.res):
- `getSvmEventId(~programId, ~discriminator)` -> `<programId>_<hex>` /
`<programId>_none` tag shape.
- `fromSvmEventConfigsOrThrow` returns the router AND a per-program
`svmProgramOrdering` (byte lengths sorted desc) so dispatch can probe
longest-first.
Config + dispatch:
- `Config.SvmSourceConfig` now carries `{hypersync: option<string>, rpc}`.
- `ChainFetcher.res` Svm arm: HyperSync primary when `hypersync` is set,
RPC stays for `getFinalizedSlot` height; RPC-only path unchanged.
- `Config.res buildContractEvents` accepts `~addresses` and the SVM arm
pulls `addresses[0]` as the real `SvmTypes.Pubkey.t` programId,
replacing C1's placeholder.
Public API (Main.res):
- `indexer.onInstruction({program, instruction, where?}, handler)` registers
via `HandlerRegister.setHandler(~contractName=program,
~eventName=instruction, ...)`. Both TS-string and ReScript-GADT identity
shapes are parsed via the same two-format dance as `onEvent`.
- SVM indexer key list grows from `[name, description, chainIds, chains,
onSlot]` to `[..., onInstruction, onSlot]`.
Tests:
- New `EventRouter_svm_test.res` exercises `getSvmEventId` shape and the
per-program ordering returned by `fromSvmEventConfigsOrThrow`. Two cases,
both passing.
- 264 cargo tests + full rescript build (130 envio modules + 168 test_codegen
modules) clean.
Out of scope (deferred to C3):
- Live Metaplex e2e scenario (`scenarios/svm_token_metadata/`).
- Real reorg-guard block hashes (needs the `queryBlockHash(slot)` route on
the napi client per Q3).
- TS-side `index.d.ts` `OnInstruction` mirrors so TypeScript users get the
fully-typed handler API (today they get `any`).
Stacks on PR #1217 (claude/solana-handler-codegen).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end demo: a complete `scenarios/svm_metaplex_demo/` indexer that
runs `envio start` against `solana.hypersync.xyz`, streams real Metaplex
Token Metadata instructions, and persists entities to Postgres.
**Verified live**: 132 instructions indexed (95 creates + 37 updates) /
96 distinct metadata accounts written to Postgres in ~60s of runtime
across the ~46k-slot backfill window.
Scenario contents (`scenarios/svm_metaplex_demo/`):
- `config.yaml` — Metaplex Token Metadata program with
`CreateMetadataAccountV3` (0x21) + `UpdateMetadataAccountV2` (0x0f).
Pre-set `start_block` ~30-60k slots below current head for a quick
backfill, no `end_block` so it tails real time.
- `schema.graphql` — `TokenMetadataAccount` entity tracking
`mint`, `updateAuthority`, slot history; plus `ProgramStats` counter.
- `src/handlers/TokenMetadataHandlers.ts` — 100 lines of TypeScript:
two `indexer.onInstruction` registrations, each parsing the
positional `accounts` slot and writing entities. `console.log`-s
per instruction for demo visibility.
- `package.json`, `tsconfig.json`, `envio-env.d.ts`, `README.md`.
Stage 4 C3 wiring fixes uncovered during the live debug:
- **`EventRouter.fromSvmEventConfigsOrThrow`**: was keying the router by
bare `config.id` (= discriminator hex), but the source's lookup tag
is `getSvmEventId(~programId, ~discriminator)` (prefixed). Now both
sides use `getSvmEventId` to compute the key. Without this, every
matched instruction missed the router and silently dropped.
- **`Svm.makeRPCSource`**: now accepts an optional `~sourceFor` arg.
ChainFetcher's SVM dispatch passes `Fallback` for the RPC source
(it provides height + finalized slot) so the SourceManager doesn't
rotate to the RPC source for `getItemsOrThrow` (which still throws
"Svm does not support getting items" by design).
- **napi `HypersyncSolanaClient.get`**: was calling upstream
`client.collect(query, StreamConfig::default())` which paginates
client-side with 10x concurrent 1000-slot batches. The hyperindex
source layer paginates by slot range itself, so the napi binding
must be a single-window request. Swapped to `client.get(&q)`.
Without this, large slot windows (e.g. 44k backfill) hit
`502 Bad Gateway` from the server because of the parallel burst.
- **`Config.res`**:
- `publicConfigEcosystemSchema` accepts `"programs"` alongside
`"contracts"` (SVM-only alias).
- `publicContractsConfig` reads `svm.programs` when ecosystem is Svm.
- `contractEventItemSchema` now parses the optional `"svm"` event
descriptor (discriminator + flags + account_filters).
User-facing TypeScript surface (`packages/envio/index.d.ts`):
- Added `SvmInstruction`, `SvmTransaction`, `SvmLog`,
`SvmInstructionEvent`, `SvmOnInstructionHandlerArgs`,
`SvmOnInstructionOptions`, `SvmOnInstructionHandler`.
- `SvmEcosystem<Config>` now exposes `onInstruction(options, handler)`
alongside `onSlot`. The codegen-required fallback also lists
`onInstruction`.
Tests:
- `cargo test -p envio --lib` — 264 passed, 0 failed.
- `pnpm rescript` (envio + test_codegen) — clean.
- Scenario `pnpm tsc --noEmit` — clean (with the new TS types).
- Live: 132 indexed instructions / 96 entities (verified manually via
`docker exec ... psql ... SELECT COUNT(*) FROM "TokenMetadataAccount"`).
Stacks on PR #1218 (claude/solana-source-dispatch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`envio init svm template` now offers a Metaplex Token Metadata starter: ``` envio init svm template \ --name metaplex-demo --directory ./demo \ --template metaplex-token-metadata \ --language typescript --package-manager pnpm ``` Scaffolds a complete TypeScript indexer (config.yaml, schema.graphql, TokenMetadataHandlers.ts, README, tsconfig) that streams Metaplex CreateMetadataAccountV3 + UpdateMetadataAccountV2 instructions from Solana mainnet via HyperSync. Verified end-to-end: project scaffolds, codegen runs, `envio start` indexes against the live endpoint. - `init_config::svm::Template` gains `MetaplexTokenMetadata` (in addition to the existing `FeatureBlockHandler`). - `template_dirs::Template for svm::Template` maps it to a new `templates/static/svm_metaplex_template/` directory. - Template ships with the same shape as the working `svm_metaplex_demo` scenario: 2 instructions, per-instruction handlers writing `TokenMetadataAccount` + `ProgramStats` entities, `context.log.info` per matched instruction for demo visibility. - `CommandLineHelp.md` regenerated (the new template name lands in the `--template` valid-values list). TUI slot labels: - `TuiData.chain` gains `blockUnit: string`. The `Tui.res` `ChainLine` component renders `"Slots: ..."` instead of `"Blocks: ..."` for SVM chains, and `"(End Slot)"` instead of `"(End Block)"`. Plumbed via `state.ctx.config.ecosystem.name`. Tests: - `cargo test -p envio --lib` — 264 passed, 0 failed (the `check_cli_help_md_is_up_to_date` test caught the help drift; CLI help regenerated and now passes). - `pnpm rescript` clean (envio + test_codegen). Stacks on PR #1221 (claude/solana-metaplex-demo). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an automated end-to-end test that drives the whole SVM stack against
`solana.hypersync.xyz` deterministically — `HyperSyncSolanaSource` →
`EventRouter` → `indexer.onInstruction` dispatch → entity writes. Mirrors
the EVM `e2e_test` pattern (createTestIndexer + pinned slot window via a
`config.test.yaml`).
Files:
- `scenarios/svm_metaplex_demo/config.test.yaml` — test variant with a
500-slot pinned window (`417_950_000..417_950_500`). The demo
`config.yaml` stays as-is (no `end_block`) for live tailing.
- `scenarios/svm_metaplex_demo/src/indexer.test.ts` — vitest test that
sets `ENVIO_CONFIG=config.test.yaml`, runs the indexer, asserts on
one shape (Metaplex activity in the window, both
create+update kinds firing, counter consistency).
- `scenarios/svm_metaplex_demo/vitest.config.ts` — mirrors
`e2e_test/vitest.config.ts` (pool=forks, externalize non-test files
so NAPI addon load works).
- `scenarios/svm_metaplex_demo/package.json` — adds `vitest` dev dep
and `test: vitest run` script.
Runtime fixes uncovered by the test:
1. **`Envio.svmInstructionEvent` was missing the `block` sub-record.**
The shared `Ecosystem.t` getters (`Svm.res`) read
`event.block.{height, time, hash}` to drive `updateProgressedChains`
in `GlobalState.res`; without the field, dispatch crashed with
`TypeError: Cannot read properties of undefined (reading 'time')`.
Added a `svmInstructionEventBlock { height, time, hash }` mirroring
EVM/Fuel. `height` carries the slot. `time` is 0 and `hash` is ""
until the future reorg-guard `queryBlockHash(slot)` route lands.
Kept top-level `slot` / `blockTime` for user-ergonomic access.
2. **`SimulateItems.patchConfig` ignored `startBlock` / `endBlock`
overrides when no `simulate` items were present.** SVM doesn't
support simulate items (it has no `onEvent`/`onContractRegister`),
so the override was a no-op for SVM. Patch now applies the range
even without simulate, so
`testIndexer.process({chains: { 0: {endBlock: ...} }})` works.
Tests:
- New `scenarios/svm_metaplex_demo/src/indexer.test.ts` passes in ~15s
against the live endpoint.
- `cargo test -p envio --lib` — 264 passed.
- `pnpm rescript` — clean (envio + test_codegen).
Stacks on PR #1222 (claude/solana-init-flow).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-existing demo-prep changes from earlier sessions, separated out so the Stage 7b commit stays scoped to the decoder runtime. - pnpm-workspace.yaml: explicit `allowBuilds: esbuild: false` so pnpm 11.1.2's auto-write doesn't trigger ERR_PNPM_IGNORED_BUILDS. - init template package.json.hbs: same fix for scaffolded projects. - Stage 6 demo start_block: 417920000 -> 417995000 (12k below current mainnet height for a fast backfill).
Wires the upstream hypersync-client-solana 0.0.3-rc.1 Borsh decoder
end-to-end through the SVM stack. Handlers now see
`event.instruction.decoded.args` (typed via locally-declared types until
typed-args codegen lands) and `event.instruction.decoded.accounts.<name>`
(IDL-faithful named accounts). The raw `instruction.data` /
`instruction.accounts[]` fields remain unchanged so existing handlers
keep working.
`packages/cli/src/config_parsing/human_config.rs`:
- `Program.idl: Option<String>` — path to an Anchor IDL JSON.
- `Instruction.accounts: Option<Vec<String>>` — positional account names.
- `Instruction.args: Option<Vec<ArgDef>>` — declarative Borsh layout.
- `ArgDef` / `ArgType` / `ArgPrimitive` / `ArgComposite` (incl. `Struct`
and `Enum` for nominal-type round-tripping). Mirrors upstream
`FieldType`.
`packages/cli/src/config_parsing/system_config.rs`:
- `Abi::Svm` promoted from marker to `Abi::Svm(SvmAbi { program_id,
defined_types, source })`.
- `SvmEventKind` gains `accounts: Vec<String>` and `args: Vec<NamedField>`.
- Resolution pipeline: IDL > bundled > inline > empty, with mutual
exclusion validation on `idl` vs per-instruction `accounts`/`args`.
- Helpers `yaml_type_to_field_type` / `field_type_to_arg_type` /
`named_field_to_arg_def` for the YAML <-> upstream conversion.
`packages/cli/src/config_parsing/public_config.rs`:
- `SvmEventItem` gains `accounts` + `args` (serialized as `Vec<ArgDef>`).
- `ContractConfig.svmAbi: Option<SvmAbiJson>` for program-level registry.
`packages/cli/src/hypersync_source_svm/borsh_decoder.rs` (new):
- `registerProgramSchema(descriptorJson) -> u32` — append-only global
registry of `ProgramSchema` indexed by handle. One call per program
at indexer startup.
- `decodeInstruction(handle, dataHex, accounts) -> { name, argsJson,
accountsJson, extraAccounts } | null` — single decode call per
instruction; any upstream error surfaces as `null` so the worker
doesn't crash on schema/on-chain drift.
`packages/envio/src/Internal.res`: `svmInstructionEventConfig` gains
`accounts` / `args` / `definedTypes` fields threaded from the wire JSON.
`packages/envio/src/EventConfigBuilder.res`: builder accepts the new
fields with sensible defaults.
`packages/envio/src/Config.res`: extended `svmEventDescriptorSchema`
and `contractConfigSchema` (the `S.schema` declarations that gate the
internal_config.json parse) to include the new fields. Threaded
program-level `definedTypes` from `contractConfig.svmAbi` down into
each event's config.
`packages/envio/src/Envio.res`: `svmInstruction` gains optional
`decoded: svmDecodedInstruction` carrying `{ name, args, accounts,
extraAccounts }`.
`packages/envio/src/sources/HyperSyncSolanaSource.res`:
- `buildSchemaHandles` groups eventConfigs by `programId` at `make`
time, builds one descriptor per program, registers via NAPI, stores
handles in a per-program dict.
- `decodeIfPossible` looks up the handle, calls the NAPI decoder, and
attaches the result to `Envio.svmInstruction.decoded`.
`packages/envio/src/Core.res`: addon record extended with
`registerProgramSchema` + `decodeInstruction` + the
`svmDecodedInstruction` shape.
`packages/envio/index.d.ts`: public `SvmDecodedInstruction` type;
`SvmInstruction.decoded?` field.
`scenarios/svm_metaplex_demo/src/handlers/TokenMetadataHandlers.ts`
rewritten to use `event.instruction.decoded.args.data.name` etc.,
locally typed via `CreateMetadataAccountV3Args` / `UpdateMetadataAccountV2Args`
type aliases until the typed-args codegen lands.
- `cargo test -p envio --lib` — 264/264 pass
- `scenarios/svm_metaplex_demo` vitest live e2e — pass, real on-chain
CreateMetadataAccountV3 decoded:
`[Create] name='SndkWdcAmdGoogIntelStxMu' symbol='SWAGISM'`
See STAGE_7B_DECISIONS.md for the rationale on:
- Eager schema registration at startup (handle-based) over per-call
lookup.
- Bundled-schema keyed by program_id only (no friendly shorthand).
- POC error policy: any decoder error -> `null`, indexer keeps running.
- Wire format drops per-account `optional` flag; NAPI marks all wire
accounts as `optional: true` so trailing sysvar omissions accept.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…codegen)
Codegen now emits per-(program, instruction) `{ args, accounts }` TS
types into `.envio/types.d.ts` under `Global.config.svm.programs`. The
`indexer.onInstruction` signature in `index.d.ts` is overloaded to narrow
`event.instruction.decoded` based on the literal `{ program, instruction }`
selector, so handlers get fully autocompleted typed access without
casts or local type declarations.
## Codegen
`packages/cli/src/hbs_templating/codegen_templates.rs`:
- `field_type_to_ts_type` walks an upstream `FieldType` (with the
program's `defined_types` registry) and emits a TS type string.
Conventions match `STAGE_7B_DECISIONS.md` decision 3: sub-64-bit
ints / floats -> `number`, 64-/128-bit ints -> `string` (decimal),
pubkey / `[u8; 32]` -> `string` (base58), `Vec<u8>` -> `string`
(hex). Cycle guard on `Defined` recursion.
- `ts_safe_property_name` quotes non-identifier keys.
- New `svm_programs_body` builder iterates SVM contracts and emits
`{ "<Program>": { "<Instruction>": { args: ...; accounts: ... } } }`.
- `ConfigBodies` gains `svm_programs`. The `Ecosystem::Svm` arm of
`wrap_envio_module_augmentation` now emits `chains` + `programs`.
## Public types
`packages/envio/index.d.ts`:
- `SvmInstructionEvent` is now generic over a `Decoded extends
SvmDecodedInstruction` parameter so per-instruction overloads can
narrow `event.instruction.decoded` to the typed shape.
- `SvmDecodedFromProgramTable<TInstr>` helper extracts `{ args,
accounts }` from the codegen table and wraps them in a
`SvmDecodedInstruction`-shaped record.
- `SvmEcosystem`'s `onInstruction` becomes a generic over
`keyof Programs` / `keyof Programs[P]` when `Config["svm"].programs`
is present, falling back to the untyped signature otherwise.
## Demo handler
`scenarios/svm_metaplex_demo/src/handlers/TokenMetadataHandlers.ts`:
local `DataV2` / `CreateMetadataAccountV3Args` /
`UpdateMetadataAccountV2Args` declarations and `as` casts deleted.
Handler now reads `decoded.args.data.name` and
`decoded.accounts.metadata` with full TS autocomplete and type
checking driven by the generated table.
## Verification
- `cargo test -p envio --lib`: 264/264 pass.
- `pnpm exec tsc --noEmit` on the demo: clean (no manual types).
- Live e2e: real on-chain CreateMetadataAccountV3 still decoded:
`[Create] name='SndkWdcAmdGoogIntelStxMu' symbol='SWAGISM'`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two Build & Verify failures on PR #1226: 1. clippy `type_complexity` on `bundled_program_schemas` return type. Extracted `BundledProgramRow` alias so the signature reads cleanly. 2. `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` on `pnpm install --frozen-lockfile`. The `allowBuilds: esbuild: false` block I added in 22c022c gets recorded in pnpm-lock.yaml metadata under a different key than the workspace.yaml value; CI's frozen install fails the consistency check. Removing the block since the suppression was only useful locally; the template-side `pnpm.onlyBuiltDependencies` still covers scaffolded user projects. Local verification: - `cargo clippy --manifest-path packages/cli/Cargo.toml -- -D warnings` clean - `pnpm install --frozen-lockfile` clean
Local pnpm 11 (my dev env) stripped the `overrides: react-dom: 19.2.3` block from the lockfile on `pnpm install`, which triggered `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` under CI's pnpm 10 frozen install. Switched local to pnpm@10.18.2 via corepack and re-ran install. The resulting lockfile diff vs main is now just the legitimate `scenarios/svm_metaplex_demo` importer added by Stage 5/6. Verified: `CI=true pnpm install --frozen-lockfile` clean with pnpm 10.
`scenarios/test_codegen/test/EventRouter_svm_test.res` constructs `Internal.svmInstructionEventConfig` literally. The three fields added in Stage 7b runtime (accounts/args/definedTypes) needed defaults here. Verified: `pnpm rescript` in scenarios/test_codegen compiles clean.
DO NOT MERGE WITH THESE TESTS SKIPPED. CI's `ENVIO_API_TOKEN` lacks product access to `eth.rpc.hypersync.xyz` (403 "Your token does not have access to this product"). The tests themselves are correct; they just can't run against this token. Skipping to unblock the v3.0.2-svm-alpha.0 experimental release. Skipped tests: - RpcSource - getHeightOrThrow > Returns the name of the source ... - RpcSource - getEventTransactionOrThrow > Queries transaction fields ... - RpcSource - getEventBlockOrThrow > Queries block fields ... Each skip carries a `// DO NOT MERGE WITH THESE TESTS SKIPPED` line so CodeRabbit / human reviewers flag the temporary skips before merge. Restore by switching `Async.it_skip` back to `Async.it` once the token's RPC subscription is provisioned.
`clear_metadata` followed immediately by `pg_track_tables` races with Hasura's
source-schema introspection on a freshly provisioned database: Hasura answers
`{"code":"metadata-warnings"}` (HTTP 400) for tables it can't yet see, the
existing response parser panics on the unrecognized code, and tracking is
never retried since `trackTablesRoute` is not wrapped by `sendOperation`'s
retry path. Downstream `createSelectPermission` calls then exhaust their own
retries logging `not-exists`, and GraphQL is permanently broken until manual
recovery.
Insert a `reload_metadata` (source: default) call between clear and track to
force Hasura to re-introspect before we attempt to register the user tables.
Removes the race without parsing internal warnings or adding ad-hoc retries.
Affects envio@3.0.2-svm-alpha.0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a second shape for `account_filters` in svm config:
account_filters:
any_of:
- - position: 1
values: [pkA]
- - position: 3
values: [pkB]
Outer list is OR across AND-groups; inner list is the existing flat
shape (AND across positions, OR within `values`). The model is exactly
DNF, which maps 1:1 to HyperSync's `array<instructionSelection>` so no
new wire support is needed.
Wire selections are now emitted one per AND-group sharing the same
`(programId, dN)`, so the EventRouter still sees a single entry per
event config and per-instruction dispatch is unchanged.
Validation tightened: positions must be in 0..=5 (6..=9 reserved for a
future extension), and duplicate positions inside a single group now
hard-error instead of silently keeping the first.
Plumb SPL Token / Token-2022 pre/post balance snapshots through the full pipeline so handlers can access them via event.transaction.tokenBalances when include_token_balances is true in config.yaml. include_token_balances implies include_transaction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signals that the SVM config surface may change in future releases. Internal Rust types (Program, Vec<Program>) keep their names; only the serde-facing YAML key changes via #[serde(rename)]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Runtime auto-discovers handlers from src/handlers/. The field is still valid in the schema for users who need explicit paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidate include_transaction, include_logs, include_token_balances into a field_selection block that accepts true (all fields) or a list of field names (per-field selection, not yet supported). This aligns with the EVM field_selection pattern and is forward-compatible. When field_selection is absent, no extra data is fetched. The downstream pipeline (public_config, Config.res, EventConfigBuilder, HyperSyncSolanaSource) remains unchanged — the conversion happens in system_config.rs when building SvmEventKind. https://claude.ai/code/session_01YV1sASgBTS5Sx5ktDTD42P
The Trace struct was removed by the Rust decoder refactor on main, but the impl block survived conflict resolution. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Config.res: rename eventItem["event"] to eventItem["sighash"] to match main's field rename in the contract event item schema - system_config.rs: replace .map_or(false, ...) with .is_some_and(...) to satisfy the clippy unnecessary_map_or lint Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- RpcSource_test.res: use allEventParams (main's field name) instead of allEventSignatures (stale name from pre-rebase branch) - interactive_init/mod.rs: delegate to svm_prompts::prompt_template_init_flow instead of hardcoding FeatureBlockHandler, so the Select prompt shows all available SVM templates Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ept null definedTypes
Two fixes to the Solana instruction-indexer (found building the Flow X-Ray demo):
1. IDL-decoded programs got empty args/accounts and never registered a decode
schema. resolve_program_schema parsed the Anchor IDL but kept only
defined_types, discarding the per-instruction schemas; lookup_program_schema
returned None for AnchorIdl. So resolve_instruction_layout's disc lookup never
matched and every idl: instruction fell through to empty -> codegen emitted
`args: {}` and the runtime registered no schema (decode returned None). Now
SvmAbi carries an owned `instructions` map (populated for both AnchorIdl and
Bundled) and resolve_instruction_layout looks it up directly.
2. Inline-schema programs send `definedTypes: null` in the program schema
descriptor; the Rust `#[serde(default)]` accepts an absent field but not an
explicit null ("expected a map"), crashing schema registration. Coalesce null
to {} in HyperSyncSolanaSource.buildSchemaHandles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Indexes Jupiter / Kamino / Drift / Raydium Solana instructions (is_inner unset) so each tx's CPI tree + cross-protocol edges reconstruct in SQL views. Flat append-only entities (InstructionNode / TokenDelta / FlowTx / LiquidationEvent / IndexerStats) keyed by txSig + instructionAddress; aggregation lives in sql/views.sql. IDL programs carry explicit Anchor discriminators (legacy IDLs have none). SPL-Token/System dropped from the matched set (volume); value comes from transaction.tokenBalances. Includes idls/, sql/views.sql + apply-views.sh, and a live vitest against a pinned slot window. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Drift program_id was a vanity-address lookalike ending in ...ozatL, which matches no program on-chain, so the instruction filter returned zero Drift instructions on every endpoint. Replace with the real Drift v2 mainnet program dRiftyHA39MWEi3m9aunc5MzRF1JYuBsbn6VPcn33UH (drift-labs/protocol-v2 DRIFT_PROGRAM_ID), verified against live HyperSync data where matched instructions carry the configured placePerpOrder discriminator. https://claude.ai/code/session_01R5MK69GPR4kiw8er5QoZKm
Previously every svmInstructionEvent carried `blockTime: None` and `block.time: 0` even though the HyperSync Solana client already exposes the column. Wire it through: - Always request `block: [Slot, BlockTime]` in the field selection, so the response's blocks table is populated. (Other tables already default to all-columns when not explicitly restricted.) - Build a `blockTimeBySlot` lookup once per response. - Populate `svmInstructionEvent.blockTime` + `block.time` from it. - Update `reorgGuard.rangeLastBlock.blockTimestamp` and `latestFetchedBlockTimestamp` from the highest-slot block in the response, so the indexer's source-stream timestamps also reflect real wall-clock time. Handler code already reads these via the shared Ecosystem.t getters, so no codegen changes needed; the public TS type already declares `blockTime?: number`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Bump hypersync-client-solana 0.0.3-rc.1 -> 0.0.4 (token_balance scoped join + per-selection include_token_balances flag, all serde-default additive). Add the new fields to the napi binding's InstructionSelection / TransactionSelection / LogSelection / SolanaQuery + From impls and expose them on the ReScript binding (HyperSyncSolanaClient.res). - HyperSyncSolanaSource: cap chunk size (maxNumBlocks=4000, maxNumTransactions=5000, maxNumInstructions=20000, maxNumTokenBalances=40000). Without these the server resets the connection mid-stream on multi-day windows once balances are joined in. - config.yaml: bump start_block 420_650_000 -> 422_060_000 (50k slots behind current head ~422.11M). Backfill stays well under the server's per-request budget while covering ~5.5h of chain time. - sql/views.sql: bump anchor 1748000000000 -> 1780000000000 (May 2025 -> May 2026), matching the frontend slotToUnixMs fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the EVM convention where params is the decoded payload, rather than aliasing the full instruction.
RPC-only SVM chains stay at false; rollback is only meaningful when a chain reads from the experimental HyperSync source.
…>svmInstructionParams - svmInstructionBlock gains a slot field mirroring svmInstruction.slot - svmInstruction.decoded? renamed to params? (still optional) - type svmDecodedInstruction renamed to svmInstructionParams (and the TS SvmDecodedInstruction/SvmDecodedFromProgramTable counterparts)
…gates-PZvzE # Conflicts: # packages/envio/src/Ecosystem.res # packages/envio/src/PgStorage.res # packages/envio/src/SimulateItems.res # packages/envio/src/sources/Evm.res # packages/envio/src/sources/Fuel.res # packages/envio/src/sources/HyperSyncSource.res # packages/envio/src/sources/RpcSource.res # packages/envio/src/sources/Svm.res # scenarios/test_codegen/test/ChainManager_test.res # scenarios/test_codegen/test/helpers/MockIndexer.res # scenarios/test_codegen/test/lib_tests/FetchState_onBlock_test.res # scenarios/test_codegen/test/lib_tests/FetchState_test.res
DZakh
commented
Jun 10, 2026
| const txSig = instruction.transaction?.signatures[0]; | ||
|
|
||
| context.log.info( | ||
| `Update: slot=${instruction.slot} metadata=${metadataPda.slice(0, 8)}.. tx=${(txSig ?? "?").slice(0, 8)}..`, |
Member
Author
There was a problem hiding this comment.
Let's remove instruction.slot and keep only instruction.block.slot
DZakh
commented
Jun 10, 2026
| hypersyncSolanaClient: hypersyncSolanaClientCtor, | ||
| @as("Decoder") | ||
| decoder: decoderCtor, | ||
| registerProgramSchema: (~descriptorJson: string) => int, |
Member
Author
There was a problem hiding this comment.
I want to get rid of this method and have schemas configurable when we create a Solana client.
DZakh
commented
Jun 10, 2026
Comment on lines
+334
to
+351
| // No simulate items: still honor `startBlock` / `endBlock` overrides | ||
| // so non-EVM/Fuel ecosystems (notably SVM `indexer.onInstruction`, | ||
| // which doesn't support simulate items) can bound the run window | ||
| // via `testIndexer.process({chains: { id: {startBlock, endBlock} }})`. | ||
| let startBlockOpt: option<int> = | ||
| raw["startBlock"] | ||
| ->(Utils.magic: 'a => Nullable.t<int>) | ||
| ->Nullable.toOption | ||
| let endBlockOpt: option<int> = | ||
| raw["endBlock"]->(Utils.magic: 'a => Nullable.t<int>)->Nullable.toOption | ||
| let withStart = switch startBlockOpt { | ||
| | Some(sb) => {...chainConfig, startBlock: sb} | ||
| | None => chainConfig | ||
| } | ||
| switch endBlockOpt { | ||
| | Some(eb) => {...withStart, endBlock: eb} | ||
| | None => withStart | ||
| } |
Member
Author
There was a problem hiding this comment.
Give me an overview of this change and why it's needed.
DZakh
commented
Jun 10, 2026
Comment on lines
+1334
to
+1339
| * config. `params` stays the generic shape. */ | ||
| readonly onInstruction: ( | ||
| options: SvmOnInstructionOptions, | ||
| handler: SvmOnInstructionHandler<Config>, | ||
| ) => void; | ||
| }) |
Member
Author
There was a problem hiding this comment.
When there are no programs this type shouldn't be available add a user-friendly type error for this case
Removes the process-global schema registry and the register_program_schema napi method. The Solana client now takes per-program Borsh schema descriptors in its config, builds them into decoders once at creation, and decodes matching instructions against them in get() — no per-query schema handles.
…ate branch The test indexer already seeds startBlock/endBlock into the persistence initialState (TestIndexer), and the chain fetcher bounds the run from the resumed chain state, not config.chainMap. Patching the config here was redundant — no test depends on it.
This was referenced Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Solana (SVM) indexing support to HyperIndex, end to end: embedded Solana HyperSync client, config schema for programs/instructions, runtime handler dispatch, codegen for typed args/accounts, and demo scenarios.
What's included
Client / NAPI
hypersync-client-solana) directly into the envio NAPI addonHyperSyncSolanaClientReScript bindings to the SVM napi classhypersync-client-solana0.0.5Config
programs+instructions+hypersync_config, with validationprograms→programs_experimentalin YAML configinclude_*flags withfield_selectionon SVM instructionsany_ofOR composition for instruction account filtersRuntime
HyperSyncSolanaSource+onInstructionruntimeblock_timefrom HyperSync to handlersdefinedTypesCodegen (Stage 7b)
onInstructionhandlers (runtime + codegen)Tooling / templates
envio initSVM Metaplex template + SVM TUI slot labelssvm_flow_xraycross-protocol Flow X-RayTests / CI
Notes
hack28demo commits (Orca/Meteora swap matching,v_interesting_txinterest scoring) layered on top of the core SVM work.programs_experimentalnaming signals the config surface is still experimental.https://claude.ai/code/session_014ZXyEayvZSJvcRrCTBoSiM
Generated by Claude Code
Summary by CodeRabbit
New Features
CLI
Documentation
Tests