Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
907a31b
agents.md: cut down AGENTS.md
aljoscha Sep 9, 2026
00557f5
doc: design decoupled coordination
aljoscha Sep 3, 2026
dc2d5b3
doc: add prompt for implementing decoupled coordination
aljoscha Sep 3, 2026
2e31582
doc: Defer nightly validation until the implementation works
aljoscha Sep 3, 2026
eb90ed3
adapter: derive sink creation from committed catalog updates
aljoscha Sep 3, 2026
f37cd8f
adapter: extract catalog index plan reconstruction
aljoscha Sep 3, 2026
fbf3a82
adapter: derive index creation from committed catalog updates
aljoscha Sep 3, 2026
6c76f64
adapter: derive MV storage registration from catalog updates
aljoscha Sep 3, 2026
3285f5d
adapter: extract catalog-based MV plan reconstruction
aljoscha Sep 3, 2026
5351ccc
test: cover uncached recovery before an MV first refresh
aljoscha Sep 3, 2026
f302cdd
doc: choose catalog-backed compaction bounds
aljoscha Sep 3, 2026
13c1821
doc: choose logical-input recovery protection
aljoscha Sep 8, 2026
a4829fa
catalog: derive logical collection inputs
aljoscha Sep 8, 2026
de8b771
doc, adapter: prune the decoupled coordination log and input-discover…
aljoscha Sep 8, 2026
8bdefbf
doc: define fresh-environment coordination milestones
aljoscha Sep 8, 2026
efbfcf8
storage: enforce explicit collection compaction bounds
aljoscha Sep 8, 2026
9f70ff4
doc: prioritize production integration in coordination work
aljoscha Sep 8, 2026
1154ec0
adapter: admit materialized views against logical inputs
aljoscha Sep 8, 2026
509287b
doc: prioritize catalog-boundary integration handoff
aljoscha Sep 8, 2026
21f525d
storage: preserve bounds across mixed create/drop batches
aljoscha Sep 8, 2026
daf0312
catalog: coordinate read protection with compaction permission
aljoscha Sep 8, 2026
995b9e3
doc: Choose durable client-scoped read protection
aljoscha Sep 8, 2026
b47e0bb
doc: Separate compaction authorization from application
aljoscha Sep 8, 2026
e45c54c
adapter: establish catalog-backed MV recovery protection
aljoscha Sep 8, 2026
ccd720a
doc: Steer coordination work toward scalable independent lifecycle
aljoscha Sep 9, 2026
f9cd1b2
doc: Clarify maintained-protection milestone coverage
aljoscha Sep 9, 2026
95afe99
doc: Defer migration work beyond decoupling milestones
aljoscha Sep 9, 2026
283d75d
doc: Consolidate coordination guidance and implementation handoffs
aljoscha Sep 10, 2026
a604a3b
doc: Settle index permission as published since and the checker boundary
aljoscha Sep 9, 2026
ae3eaf8
catalog: Integrate sparse recovery protection and changed-record publ…
aljoscha Sep 10, 2026
e606fc1
prompt: tell to re-read prompt when compaction happens
aljoscha Sep 9, 2026
cc2bac5
test: Bound milestone publication measurements to 1,000 objects
aljoscha Sep 10, 2026
cbf055e
test: Document catalog payload metrics and close bounded recovery evi…
aljoscha Sep 10, 2026
a89eb08
catalog: prepare maintained lifetimes independently of installation
aljoscha Sep 10, 2026
215bb46
doc: Decide lifecycle placement and the query client for milestone 2
aljoscha Sep 10, 2026
8328f7f
catalog: preserve materialized-view read versions across replacement
aljoscha Sep 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 1 addition & 157 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,5 @@
# Materialize

## Skills

Canonical agent skills in `.agents/skills/`. `.claude/skills` is compat symlink
for Claude Code. Check `mz-*` skill before tasks — encodes project conventions,
saves time.

Use the `mz-test` skill before running ANY tests, even mid-task — the canonical
commands aren't the obvious ones (e.g. `bin/sqllogictest --optimized`, not
`cargo build --bin sqllogictest`).

Use the `mz-run` skill before building, running, formatting, or linting. `bin/fmt` and `bin/lint` are the canonical entry points, NOT `cargo fmt`, `rustfmt`, or a bare `cargo clippy`. `bin/environmentd`, not `cargo build --bin environmentd`.

Use the `mz-commit` skill before `git commit`, `git push`, or `gh pr create`.

Use the `mz-debug-ci` skill before the first `bk` or `gh pr checks` command, or when handed a Buildkite URL.

## Code navigation

For operation flow tracing, read first:

* `doc/developer/generated/flows.md` — maps operations (query lifecycle, source ingestion, MV creation, sink lifecycle, catalog DDL, timestamp selection, persist read/write, controller architecture) to `crate::module` paths in execution order.
* `doc/developer/generated/<crate>/_crate.md` — per-crate overview: modules, key types, dependencies.
* `doc/developer/generated/<crate>/<module>.md` — per-file docs.

> **READ-ONLY: `doc/developer/generated/` is generated, not authored.**
> The entire `doc/developer/generated/` tree is maintained exclusively by the
> recurring documentation agent, which runs the `update-docs` skill/command.
> That agent is the *only* session permitted to create, edit, or delete files
> under this directory.
>
> In any other session: **treat `doc/developer/generated/` as read-only.** Use
> it for navigation and context, but never edit, create, delete, or regenerate
> files there — not even to "fix" something you noticed, and not as part of an
> unrelated change. These files carry `source`/`revision` front-matter that the
> recurring agent manages; hand edits desync that bookkeeping. If a generated
> doc is wrong or stale, report it in your response rather than editing it, and
> leave the correction to the `update-docs` agent. Do not stage or commit any
> path under `doc/developer/generated/` unless you are explicitly running the
> `update-docs` workflow.

## Dependency management

### Workspace dependencies
Expand All @@ -62,122 +22,6 @@ Never regenerate full Cargo.lock. When changing deps:
* **Never bare `cargo update`** — bumps every semver-compatible dep, causes unrelated breakage from transitive changes.
* **If lock regenerated**, diff before commit (`git diff Cargo.lock | grep '^[+-]version'`) and pin back unintended bumps with `cargo update -p <crate> --precise <old-version>`.

### Licensing
## Licensing

Two files control license policy, **keep in sync**: `deny.toml` (`[licenses].allow`) and `about.toml` (`accepted`). New dep with new license not already allowed: add SPDX identifier to both.

## Guidance

* When designing specs or implementing features, preserve the full scope and
capability of the solution. Do not substitute dynamic or generative
approaches with hardcoded data, skip automation that a reference
implementation provides, or simplify away the parts that make a feature
robust and maintainable. If a reference codebase generates data from an
authoritative source, our implementation should do the same, not ship a
static snapshot. When you feel tempted to reduce scope or take a shortcut,
flag it to the user and workshop an alternative together rather than silently
downgrading the design.
* When making code changes, run cheap checkers/linters and formatters before
reporting success and/or committing changes. For Rust, these would be
`bin/fmt` and `cargo check`.
* When debugging CI or lint failures, start by reproducing the exact failing
command locally and reading its output. Do not run generic checks (clippy,
fmt, grep) in a shotgun approach.
* We value simplicity and clear abstractions. We especially care about
designing the interfaces or boundaries between components well. This includes
components, traits, interfaces, modules, and crates.
* In code comments or inline documentation, we value clearly described
contracts and assumptions.
* In code comments, we don't like "fluff" comments, comments that describe what
code does when it is obvious from the code. Good code should be readable. It
_is_ okay to call out tricky parts of the code or "nota benes".
* In code comments and code documentation we value concise but complete
comments.
* In code comments and documentation, don't refer to potential previous states
of the code, or things like future PRs, try not to use chronology in there,
except when it's needed to explain why a certain thing behaves as it does and
we need to record that knowledge. In general comments need to stand on their
own and make sense from just looking at them and the code around it, not
previous changes.
* Avoid em-dashes for structuring sentences in any prose: code comments, specs,
design docs, all of it. Restructure with full stops and commas instead. The
same goes for semicolons, though one is fine where splitting would mangle the
sentence.
* Our guidance applies both when writing new code or designs, or when we notice
deviations in code or architecture that we are working on. At the same time,
we want to keep our changes minimal so it's good to call out deviations and
then we can decide together what to do about it.
* Never put a side effect inside `debug_assert!` (or `debug_assert_eq!`, etc.).
The macro body compiles out whenever debug-assertions are off, which includes
`[profile.optimized]` (what `bin/environmentd` and mzcompose use) and
`[profile.release]`. Only `[profile.ci]` turns them on. A side effect written
there, for example `debug_assert!(map.insert(k, v).is_none())`, runs under
`cargo test` but silently vanishes in optimized and release builds, leaving
the logic it powered inert and profiler-blind. Bind the effect to a `let`
outside the assert, then assert on the bound value. Clippy's
`debug_assert_with_mut_call` catches `&mut self` receivers but not `&self` or
free-function side effects.
* Never write vendor, customer, or account names into durable or user-facing
surfaces: committed code, comments, column comments, docs, specs, commit
messages, PR bodies, or test fixtures. Anything persisted to disk gets shared,
indexed, and outlives the context, so a name in it is a leak. Anonymize to the
technical pattern instead, for example "a wide unfiltered LEFT JOIN driving a
freshness incident" rather than who hit it. Names are fine in ephemeral chat,
not on disk.
* Never use `std::collections::HashMap`/`HashSet` directly, `clippy.toml`'s
`disallowed-types` blocks it. Use `BTreeMap`/`BTreeSet` when iteration order
matters, or `mz_ore::collections::HashMap` for keyed-only access with no
iteration. Hash-order iteration is a real nondeterminism source, for example
an unstable order reaching plan output or persisted state, not a style nit.
* A new feature flag should default off in production but default ON in the
test/CI configuration, so the new code path is exercised by sqllogictest,
testdrive, and optimizer goldens before it earns trust. Production safety and
test coverage are separate settings. Wire the override through
`system_parameter_default`: the `--system-parameter-default=NAME=VALUE` CLI
flag (env `SYSTEM_PARAMETER_DEFAULT`) for sqllogictest and environmentd
binaries, or `TestHarness::with_system_parameter_default` for Rust
integration tests.

## Code comments

Specifics for code comments and documentation:

Spend comments on the non-obvious: concurrency and async hazards (races,
lease/handle expiry, values that must not be held across an await point),
ordering constraints ("X must happen before Y, else Z"), invariants whose
violation panics or corrupts data, restart/recovery semantics, the origin of
magic constants, and why the obvious alternative was not taken. Idiomatic code
(match arms, iterator chains, getters, logging) needs none.

A doc comment is the caller's contract: a one-sentence summary, then only the
invariants and semantics a caller must know. A self-evident public item needs
only that one line. Don't narrate the body in rustdoc ("Phase 1 ... Phase
2 ...", or enumerating a flag's branches). Reasoning about how or why the code
works goes in an inline `//` at the decision point, or in a module-level `//!`
when it is about how the pieces fit together. Document a struct field only when
its meaning is subtle.

Every fact gets exactly one owning comment: the decision point for reasoning,
the mechanism for mechanism docs, the public setter or constant for config
semantics. Everywhere else either points at the owner or says nothing. The
tell is the same clause appearing near-verbatim in a module doc, a struct doc,
and an inline comment; when that happens, pick the owner and cut the rest. In
particular: don't restate a callee's documented contract at its call sites,
don't narrate in one function's doc what a sibling's doc already owns, and
don't duplicate an assert message in a comment on the same line.

Budget doc-comment paragraphs by decisions: each paragraph should carry a
distinct decision or invariant, not re-argue one documented elsewhere or
enumerate the item's callers or fields. Skip speculative comments about what
a future implementation could do (one `TODO` at the owning seam is enough),
and skip comments explaining an absence when the surrounding docs already
imply it. No ASCII section-divider banners (`// ----- helpers -----`); item
placement carries the structure. Performance claims in comments should state
a constraint or a measured number, not unverifiable color ("the compiler can
autovectorize this").

The same economy applies to tests: a test whose name and assert messages
state the property needs no doc comment. Keep test docs for non-obvious
setup, fixtures whose shape encodes the scenario, and multi-phase protocols.

Mark counterintuitive gotchas with `NOTE:` and future work with `TODO:`.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading