Per-entity ClickHouse table tuning via @storage(clickhouse: {...})

[moose-code]

Implements the per-entity ClickHouse table tuning proposal from #1409, with the API shape requested in the issue thread:

type Transfer @storage(clickhouse: {
  partitionBy: "toYYYYMM(timestamp)",
  orderBy: ["timestamp"],
  ttl: "timestamp + INTERVAL 2 YEAR"
}) {
  id: ID!
  timestamp: Timestamp!
}

What changed

The @storage directive's clickhouse arg now accepts a table options object in addition to a boolean. The object form implies the backend is enabled, and every key is optional:

Option	Type	Effect on envio_history_<Entity> DDL
partitionBy	String	ClickHouse expression emitted as PARTITION BY <expr>; bare identifiers that match entity fields are resolved to the ClickHouse column names
orderBy	[String!]	Entity field names leading the sorting key in place of the default id prefix; envio_checkpoint_id always stays appended, so the key becomes ORDER BY (<fields...>, envio_checkpoint_id)
ttl	String	ClickHouse expression emitted as TTL <expr>, with the same field-name resolution as partitionBy

Generated DDL for the example above:

CREATE TABLE IF NOT EXISTS envio.`envio_history_Transfer` (
  `id` String,
  `timestamp` DateTime64(3, 'UTC'),
  `envio_checkpoint_id` UInt64,
  `envio_change` Enum8('SET', 'DELETE')
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(`timestamp`)
ORDER BY (`timestamp`, envio_checkpoint_id)
TTL `timestamp` + INTERVAL 2 YEAR

Design notes

• Validated at codegen (opt-in per entity, as proposed in the issue): orderBy entries must be existing entity fields written as in the schema, and must be non-nullable, non-array, non-derived. id is rejected (it's the default sorting key), and so are BigInt/BigDecimal fields (stored as String in ClickHouse, so ordering would be lexicographic). Unknown option keys, empty expressions, and empty/duplicate orderBy lists are rejected too.
• Field names resolve to ClickHouse columns at DDL time: orderBy entries and bare identifiers inside partitionBy/ttl expressions are written as schema field names and rewritten to the actual columns, so entity references (token → token_id) and column_name_format: snake_case renames keep working. Functions, keywords, numbers, string literals and already-backticked identifiers pass through untouched. Invalid expressions still fail loudly at storage initialization.
• envio_checkpoint_id is always kept in the sorting key (per review discussion): a custom orderBy sets the leading columns, and the checkpoint id stays appended for a deterministic tie-break and a clean ascending run per prefix for the current-state view's dedup. id is intentionally not kept — ClickHouse entities are read-only, so no path looks history rows up by id.
• Lifecycle: the options travel through the persisted public config JSON mirroring the directive shape (storage.clickhouse is bool | object), so changing them diffs against the stored config on restart and prompts a reset — a MergeTree table's PARTITION BY/ORDER BY can't be altered in place for existing data. Booleans keep serializing exactly as before (an empty clickhouse: {} normalizes to the boolean form), so existing projects' persisted configs stay byte-identical.
• The dedup entity view is unaffected: its ORDER BY ... LIMIT 1 BY id is a query-time sort, independent of the table's sorting key. A custom sorting key mainly buys pruning for time-bounded queries against envio_history_* (and TTL gives the bounded-retention story from the issue).

Not in this PR (follow-ups from #1409)

• Lifecycle-managed materialized views (incl. recentWindow sugar and isRealtime gating)
• Append-only/immutability hint for a cheaper serving view

Happy to tackle those next — they're meatier lifecycle features and deserve their own PRs.

Testing

• Rust unit tests for directive parsing, option/orderBy validation errors, storage routing validation, and the public-config JSON shape.
• ReScript tests driving the full pipeline (YAML + schema → real Rust parser via NAPI → Config.res → DDL): options reach entityConfig.storage and the generated CREATE TABLE, including snake_case + linked-entity resolution in orderBy and partitionBy/ttl expressions.

Closes #1409's first proposal (per-entity table options).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwrS1UjDN5bZYrM6sZdkY5

---

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Added per-entity ClickHouse table configuration with partitioning, ordering, and TTL options.
  • Supports configuring ClickHouse storage with either a simple enabled/disabled value or detailed table options.
  • Generated history tables now apply the configured ClickHouse options.

• Bug Fixes

  • Improved validation for ordering fields, including existence, nullability, arrays, and duplicates.
  • Enhanced configuration errors for unsupported or invalid ClickHouse options.

• Tests

  • Added coverage for ClickHouse options, validation rules, configuration parsing, and generated SQL.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Per-entity ClickHouse table options

Layer / File(s)	Summary
Directive parsing and validation packages/cli/src/config_parsing/entity_parsing.rs	ClickHouse storage accepts booleans or options objects containing partitionBy, orderBy, and ttl, with validation for option shapes and sorting fields.
Configuration serialization and runtime storage shape packages/cli/src/config_parsing/public_config.rs, packages/cli/src/config_parsing/system_config.rs, packages/envio/src/Config.res, packages/envio/src/Internal.res	ClickHouse options are serialized, parsed into runtime entity storage, and included in backend enablement validation.
ClickHouse history-table DDL packages/envio/src/bindings/ClickHouse.res	History-table SQL now emits configured PARTITION BY, resolved ORDER BY, and TTL clauses.
Integration fixtures and validation coverage packages/cli/test/..., packages/envio-tests/test/ConfigYaml_test.res, packages/cli/src/hbs_templating/codegen_templates.rs, scenarios/test_codegen/test/lib_tests/ClickHouse_test.res	Fixtures and tests cover option propagation, invalid directives, field resolution, and generated SQL.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GraphQLSchema
  participant CLIConfig
  participant RuntimeConfig
  participant ClickHouseDDL
  GraphQLSchema->>CLIConfig: define per-entity ClickHouse options
  CLIConfig->>RuntimeConfig: serialize and parse storage options
  RuntimeConfig->>ClickHouseDDL: provide clickhouseOptions
  ClickHouseDDL->>ClickHouseDDL: resolve orderBy columns
  ClickHouseDDL->>ClickHouseDDL: generate partition, order, and TTL clauses

Loading

Possibly related PRs

• enviodev/hyperindex#1194: Modifies the same per-entity @storage parsing and validation flow.
• enviodev/hyperindex#1297: Overlaps with the storage validation and public configuration emission paths.
• enviodev/hyperindex#1300: Modifies ClickHouse backend enablement and validation paths.

Suggested reviewers: dzakh

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: per-entity ClickHouse table tuning via the storage directive.
Linked Issues check	✅ Passed	The PR implements #1409's per-entity PARTITION BY, ORDER BY, and TTL options with validation and config persistence; follow-up MV work is correctly excluded.
Out of Scope Changes check	✅ Passed	The changes stay focused on ClickHouse storage option parsing, validation, config plumbing, DDL generation, and tests, with no unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

packages/envio/src/bindings/ClickHouse.res (1)

365-387: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Avoid row-deleting TTL on the history table. The current-state view picks the latest surviving history row per entity, then filters to SET; expiring a DELETE tombstone can resurrect an older SET, and expiring the only latest row can hide the entity entirely. If retention is required, keep current state separately or preserve each entity’s latest row/tombstone.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/envio/src/bindings/ClickHouse.res` around lines 365 - 387, Remove
the TTL clause from the history table DDL generated by the surrounding
table-creation function, including the ttl interpolation represented by
ttlClause. Do not apply row-expiring retention to history rows unless the
implementation separately preserves each entity’s latest SET or DELETE
tombstone; keep the existing schema and ordering behavior unchanged.

🧹 Nitpick comments (1)

packages/envio/src/Config.res (1)

319-320: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove narration-only comments from these type declarations.

• packages/envio/src/Config.res#L319-L320: remove the comment; the union already shows the boolean/object forms.
• packages/envio/src/Internal.res#L723-L724: remove the provenance/consumer comment; the fields are self-explanatory.

As per coding guidelines: “Don't write a comment that restates what the code already says” and “Never narrate the refactor itself.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/envio/src/Config.res` around lines 319 - 320, Remove the
narration-only comment above the entity’s clickhouse storage type declaration in
packages/envio/src/Config.res at lines 319-320; the union type is
self-explanatory. Also remove the provenance/consumer comment in
packages/envio/src/Internal.res at lines 723-724, leaving the existing
self-describing fields unchanged.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@packages/envio/src/bindings/ClickHouse.res`:
- Around line 365-387: Remove the TTL clause from the history table DDL
generated by the surrounding table-creation function, including the ttl
interpolation represented by ttlClause. Do not apply row-expiring retention to
history rows unless the implementation separately preserves each entity’s latest
SET or DELETE tombstone; keep the existing schema and ordering behavior
unchanged.

---

Nitpick comments:
In `@packages/envio/src/Config.res`:
- Around line 319-320: Remove the narration-only comment above the entity’s
clickhouse storage type declaration in packages/envio/src/Config.res at lines
319-320; the union type is self-explanatory. Also remove the provenance/consumer
comment in packages/envio/src/Internal.res at lines 723-724, leaving the
existing self-describing fields unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2fb3849b-d847-42a6-8c91-dca26982d898

📥 Commits

Reviewing files that changed from the base of the PR and between a28ee98 and ecc3191.

📒 Files selected for processing (11)

• packages/cli/src/config_parsing/entity_parsing.rs
• packages/cli/src/config_parsing/public_config.rs
• packages/cli/src/config_parsing/system_config.rs
• packages/cli/src/hbs_templating/codegen_templates.rs
• packages/cli/test/configs/config-clickhouse-options.yaml
• packages/cli/test/schemas/schema-with-clickhouse-options.graphql
• packages/envio/src/Config.res
• packages/envio/src/Internal.res
• packages/envio/src/bindings/ClickHouse.res
• scenarios/test_codegen/test/ConfigYaml_test.res
• scenarios/test_codegen/test/lib_tests/ClickHouse_test.res

[DZakh]

We need the (${Table.idFieldName}, ${EntityHistory.checkpointIdFieldName}) for internal HyperIndex write/read logic. I wonder whether the complete overwrite will be harmful to indexing performance.

[moose-code]

Checked what actually touches the history table today, to size the risk:

• Writes (setUpdatesOrThrow) are plain batch INSERTs — no reads, and the part-sort cost at insert/merge time is the same for either key.
• Rollback (resume) runs ALTER TABLE … DELETE WHERE envio_checkpoint_id > X. Since envio_checkpoint_id isn't a leading column of the current (id, envio_checkpoint_id) key either, that mutation scans parts regardless of the key.
• The entity view sorts by envio_checkpoint_id DESC LIMIT 1 BY id at query time and gets no read-in-order benefit from the current key, so its cost is unchanged by the overwrite.

So no internal path depends on the sorting key today. But if you want to keep the door open for logic that would (e.g. per-id stale-history pruning), there's a zero-downside variant:

Option B: orderBy sets the leading key columns and the internals are always appended — orderBy: ["timestamp"] → ORDER BY (timestamp, id, envio_checkpoint_id). Time-range pruning only needs the prefix, so the user-facing benefit is identical, and (id, envio_checkpoint_id) stays in the key. Cost is just a slightly wider per-insert sort.

Option A (current PR): the key is exactly what the user writes.

Happy to go either way — B is a small change, say the word and I'll push it.

---

Generated by Claude Code

[DZakh]

The implementation looks good. Just one concern about the behavior change.