doc: design decoupled coordination

[aljoscha]

Motivation

Make the catalog authoritative for maintained storage and compute lifecycle,
independent of the adapter that initiated a change. Query-local execution stays
on a separate fast protocol. The design records the ownership, read-protection,
recovery, and query-ordering boundaries.

Description

Milestone 1 is complete within the agreed 100/1,000-object performance scope.
The implementation, production evidence, and final regular PR CI are verified.
The target is fresh environments and same-version recovery.

• Collection compaction permission and logical MV requirements commit together.
  Admission accounts for optimized-away inputs and committed permission, not
  lagging physical compaction. Pending first refreshes and builtin inputs are protected.
• Source and sink requirements derive from catalog definitions and durable progress,
  including remap protection, source-export birth floors, and sink input alteration.
  Fresh sinks use committed input permission for their automatic cutoff, including
  SNAPSHOT = false. Existing pending output remains protected.
• Batched publication advances requirements with durable output progress and includes
  storage and compute compaction proposals. Controller proposals preserve creation
  and execution holds. Committed bounds govern storage and readable compute exports.
• Index bounds appear at first publication. Installation does not write the catalog
  or wait for the writer. Recovery respects feasible saved permission and replaces
  at actual input readability when the saved bound is stale conservatism. Live
  execution holds remain protected.
• Publication consumes changed controller records and indexed maintained requirements.
  Durable validation visits affected records and consumers. Sparse index retirement
  follows final lifetime changes, without MIN birth records or whole-index scans.
  General catalog snapshot and storage-metadata cloning costs remain for measurement.
• The bound subscriber delivers changed records independently of the SQL savepoint.
  It is named for that scope, not presented as a maintained-lifecycle executor.
• The existing catalog diagnostic endpoint can reconstruct independent durable input
  at an aligned prefix off the coordinator thread. Normal invariant checks retain
  their behavior. Testdrive requests an explicit reconstruction response and retains
  external comparison only for older servers. There is no frozen catalog mode,
  dedicated dump-snapshot command, or upper header. Publication cadence is positive.

The adapter stages protection through one operation. Storage creation and emitted
sink-command tests cover execution holds preserving history older than policy
permission, with and without catalog governance.

Protection defaults off in production and on in tests. Existing unprotected
objects are not converted. Sink exports and indexes install through catalog
implications. MV storage registration follows implications, but MV compute
installation still uses a sequencer closure. Controllers still live in the adapter.
Metric sinks require input execution protection, not historical output guarantees.

The restart demo covers actual persist compaction, an optimizer-eliminated input,
uncached pending-refresh recovery, system-catalog inputs, source/remap recovery,
pending sink output, and the fresh no-snapshot sink cutoff. A separate configurable
workflow measures production publication, concurrent DDL latency, catalog traffic,
and retained history at increasing collection counts. The completed runs and their limits are recorded below.

The bounded measurement scope is 100 and 1,000 generated objects, retaining the
shared-view index topology and diagnostics. Full-scale performance work is deferred.
The history consolidates the older WIP implementation while preserving the design
decision and prompt-only changes as distinct patches. Temporary experiment dispatch
is removed from the final tree, but the workflows remain runnable.

Later milestones cover
independent maintained-lifecycle following, independent query clients and durable
client protection, and catalog/query ordering. The permission subscriber is not yet
an independent maintained-lifecycle executor. Existing-environment conversion,
builtin schema migration, and protected-environment version upgrades remain
follow-on work after fresh-environment decoupling.

Milestone 2 prerequisites

Commit a89eb08f makes MV requirement completeness a catalog-writer responsibility.
Creation without a sequencer-supplied requirement protects all logical query inputs,
including optimized-away reads. Optional selected frontiers are checked against
final-state birth promises. Replacement targets are distinct from query reads, and
retired writer aliases keep completed requirements rather than adopting a new query.

Storage metadata preparation uses final catalog membership, including all collection
versions, rather than installed collections. Shared live aliases prevent shard
retirement, while orphan mappings do not. Preparation does not physically finalize
shards. No new durable records or test-only production interfaces are added.

Commit 8328f7f1 preserves user-MV collection versions in stored SQL and in
session-catalog GlobalId lookups. Downstream SQL and recovery requirements keep the
same input identities across replacements, including consumers created between
successive replacements. Exact requirement validation is unchanged.

Milestone 2 is not complete. The agreed order is an in-process adapter query client
with durable client protection, the cluster-side lifecycle/query connection split,
cooperating catalog writers, then committed-state MV/metric-sink installation and sink
alteration followed by the controller process move. Adapters retain DDL and table
appends. The lifecycle process only follows the catalog, enacts maintained state,
and publishes protection. No controller-access API or DDL forwarding through it is
planned. Protection record shape and incarnation reclamation are awaiting approval
before implementation. The cooperating-writer mechanism has its own later approval gate.

Validation

Current head 8328f7f1 is awaiting regular PR CI in
build 134383.
The prerequisite head a89eb08f failed Testdrive 5 in
build 134273. Upstream MV
replacement rebound a downstream MV's SQL input to a new GlobalId while its durable
requirement retained the old one, blocking publication. The focused Rust reproduction
failed twice before the version-binding correction and passes with it. The kept
regression covers two replacements, an intervening new consumer, unchanged protected
read identities, successful publication, and both public GlobalId lookup APIs.

On the current source, bin/fmt, git diff --check,
cargo test -p mz-adapter --lib test_materialized_view_, and
cargo clippy --all-targets -- -D warnings pass. The two targeted tests are the
birth-admission and downstream-replacement regressions. These local results do not
replace pending regular PR acceptance.
The earlier build 134269
failed its two Clippy jobs on eight unwrap() uses in the new adapter test.
Those now have descriptive expect() messages. The correction is folded into the
single prerequisite commit, with no production or assertion changes.

Local verification passed:

• bin/fmt and git diff --check.
• cargo clippy --all-targets -- -D warnings, after reproducing the CI failure.
• Three storage preparation tests and the bootstrap stale-WAL recovery test.
• cargo test -p mz-adapter --lib test_materialized_view_birth_admission.

The MV test exercises writer-derived requirements, optimized-away inputs, final-state
admission, refresh promises, replacement cutover, and retired-writer aliases against
an in-memory Persist catalog. The three storage preparation tests fail at the original
installed-collection lookup. The MV test fails because its requirement is absent when
writer derivation and its validation are removed. Both pass with their implementations
restored. These are prerequisite boundary proofs, not a maintained-lifecycle subscriber
demo. The milestone 1 acceptance and production evidence below apply to their recorded
heads, not to this prerequisite commit.

The six schema-3 publication jobs passed at eac2b075a567 in
build 134248.
Final regular CI passed for cleanup head cbf055e2 in
build 134250.
Its production and test source trees exactly match the evidence head. The PR remains a draft and nightly is
not enabled. The two earlier lint failures were the missing generated inventory
entries for the new payload metrics. Those entries are included, and both lint
jobs pass in the evidence run.

Native targeted coverage includes sparse durable bounds, independent diagnostic
reconstruction, held-readability sink commands, controller compaction, and exact
committed-payload accounting. The latter checks insertions, replacements,
retractions, unchanged records, and rejected/read-only/savepoint writes.

Recovery demonstration

Restart 2 passed in build 134248. It exercises the production path with expression caching
disabled, actual persist batch compaction, and an optimizer-eliminated input.
At the pending-refresh observation:

• Pending refresh: 1789047257958.
• Control compacted batch since: 1789047280000, beyond that refresh.
• The protected eliminated input compacted at the pending refresh and retained
  147 updates, versus 81 in the identically revised control.
• Its current-state batch footprint was 592 bytes larger than the control.
  This is one small-scenario footprint observation, not total disk accounting.
• An MV at permission minus one was rejected while physical since was still
  below that timestamp, demonstrating admission by permission rather than lagging
  physical compaction.

The assertions recover without a replica, resume the pending MV to return 1
while the live input returns 2, recover the system-catalog-input MV, and release
old input history after completion without dropping the MVs. Ongoing output
advances its requirement. The workflow also covers source/remap and sink recovery,
no-snapshot sink admission, and unpublished/published index recovery.

Read-only following

All four proof jobs passed at 44a20e37 in
build 134190.
Paired values are before/after sibling drop. They measure from observed writer
permission to observed follower frontier, including SQL observation, not true
commit-to-apply time or isolated subscriber CPU.

Requested indexes / cadence	Actual cohort before → after drop	Observed all-complete lag, seconds
1 / 1s	2 → 1	0.845 / 0.894
100 / 1s	101 → 100	0.610 / 0.981
1,000 / 1s	1,001 → 1,000	0.527 / 0.242
1,000 / 10s	1,001 → 1,000	1.506 / 1.522

The writer snapshot requests took 1.36–1.75 seconds. These are individual
observations, not latency distributions. The proof also covers unpublished bounds,
following beside a frozen SQL savepoint, suspended-cluster prewarming, capped
permission, and advancement after restoring cadence. The bootstrap DROP overlaps
startup but does not force an exact bootstrap phase.

Bounded publication measurements

All six jobs in build 134248 completed both sizes, each with three 30-second
observation rounds. Sparse means ten advancing tables plus 90/990 unchanged
governed indexes and a support view. Active means 100/1,000 advancing tables.
Catalog-wide counters include builtin work, not just the generated objects.

Decimal kB = 1,000 bytes. Rate ranges below are the envelope of each round's
outer/inner scrape-time bounds, not confidence intervals. Pair latencies are
successful CREATE/DROP execution latencies, not scheduled-arrival response times.

Control	Bound packed-row kB/s, 100 objects	Bound packed-row kB/s, 1,000 objects	DDL pairs completed/offered at 1,000	Pair p95 ms at 1,000	Persist metadata kB/s at 1,000
Sparse, 1s, no DDL/observer	87.21–90.52	87.94–90.81	0/0	N/A	6.65–7.03
Sparse, 1s, DDL, no observer	87.13–88.48	84.93–88.37	180/180	100.3–104.6	35.06–39.30
Sparse, 1s, DDL, polling	82.53–87.40	82.26–88.16	168/180	451.2–494.9	36.38–36.94
Sparse, 10s, DDL, no observer	7.89–9.17	9.03–9.14	180/180	105.3–111.3	32.42–33.25
Active, 1s, DDL, no observer	108.45–112.85	344.20–349.29	180/180	151.9–162.1	37.00–37.96
Active, 10s, DDL, no observer	10.55–11.76	37.18–37.51	180/180	50.7–59.0	32.33–34.48

At 1,000 objects, maintained-requirement payload adds 14.6–16.0 kB/s at 1s cadence
or 1.57–1.60 kB/s at 10s. Other catalog rows add about 4.8 kB/s with the full DDL
load, and zero without DDL. A bound replacement normally contributes two row
updates, not one publication.

All non-observer DDL runs achieved 2 pairs/s with no missed slots. Polling completed
172/180 pairs at 100 objects and 168/180 at 1,000, missing 8 and 12 slots. No DDL
request straddled the observation end. At 100 objects, sparse no-observer p95 was
69.3–79.3 ms versus 100.3–104.6 ms at 1,000. The corresponding active values were
65.0–77.4 ms and 151.9–162.1 ms. This is bounded scaling evidence, not proof of a
complexity class or no regression against a pre-feature baseline.

Findings:

• Adding unchanged governed indexes leaves emitted bound payload approximately
  flat. Increasing advancing tables increases it. Baseline catalog cloning still
  exists, so flat payload is not a claim of constant total coordinator work.
• The 10s cadence cuts protection payload substantially in these windows, while
  combined consensus metadata with DDL remains about 32–40 kB/s. This does not
  establish a universal cadence ratio. The default cadence is unchanged.
• Polling is materially perturbative. Lower traffic with fewer completed DDL pairs
  is not an efficiency improvement.
• Setup for the 1,000-object cohorts took 47–66 seconds, versus 9–16 seconds for
  100. The 10,000-identical-index setup failures in build 134190 remain documented:
  30–104-second coordinator stalls, quadratic notice/dependency costs, and repeated
  uncached optimization. The exact stall attribution was not profiled. Larger-scale
  work is deferred by agreement, not hidden by weakening diagnostics.

Packed-row bytes exclude timestamps, diff encoding, compression, and network framing.
Persist state-diff bytes measure encoded consensus metadata, not row payload or all
network/blob traffic. Counter samples can include boundary activity, so dividing
these numerators by in-window DDL count does not yield exact bytes per DDL.

Current-state footprint and sampled history

These are first-round-before to last-round-after metric scrapes for 1,000 objects,
including intervening update phases. They are not total retained disk bytes or
measurements of garbage-collection completion.

Control	Catalog current-state batch bytes, before → after
Sparse, no DDL, 1s	13,820,768 → 20,378,878
Sparse, DDL, 1s	14,864,649 → 20,205,642
Sparse, DDL, 10s	4,344,170 → 5,272,171
Active, DDL, 1s	31,224,602 → 57,026,151
Active, DDL, 10s	5,700,468 → 8,377,908

Every sampled table had 2,640 referenced batch bytes and two parts at both metric
endpoints. Final sampled retained-update counts were three. INSPECT snapshots and
byte scrapes are sequential, not synchronized. Catalog footprints include setup
and startup history as well as live records, so these are not isolated retention
savings or fleet-wide estimates.

Reproduction

Use disposable CI composition projects, not the development VM:

bin/mzcompose --find restart --project-name coord-m1-demo run catalog-read-protection
bin/mzcompose --find 0dt --project-name coord-m1-follower run index-read-protection
bin/mzcompose --find restart --project-name coord-m1-measure run catalog-publication-measurement \
  --collection-counts 100,1000 --publication-rounds 3 --observation-seconds 30 \
  --publication-interval-ms 1000 --timeout-seconds 300 --observer none --ddl-rate-hz 2

These workflows reset their project volumes. Sparse controls add
--active-collections 10 --filler-kind indexes. The no-DDL control uses
--ddl-rate-hz 0, polling uses --observer poll, and slow cadence uses
--publication-interval-ms 10000.

Replaces #38694 under the outcome-focused name Decoupled coordination.

[aljoscha]

Commit 9d5e1a9 connects storage-backed catalog read protection to committed delivery. Bounds and maintained requirements have separate durable records, final-state transaction validation rejects incompatibility and regression, and the adapter returns admission errors before its fatal commit boundary. Committed bounds reach storage through implications. Dry runs model collection births and drops without controller effects.

This is an intermediate milestone 1 change, not activation. SQL creation still needs to produce initial bounds and logical-input requirements with secured readability, including shared-shard initialization. Recovery and durable-progress publication remain unwired.

Validation: formatting, targeted compilation, and the durable admission/reopen and adapter admission/dry-run tests pass. Independent review findings are resolved. Regular PR CI is still pending: https://buildkite.com/materialize/test/builds/134033