feat(supervisor)!: apply streamed configuration snapshots

[pimlock]

Summary

Initialize supervisors and apply live configuration updates through ConnectSupervisor when the gateway runs with config_delivery_mode = "push". Supervisors advertise supports_config_apply; the gateway answers with config_apply_enabled, requires a successful bootstrap, and marks the supervisor ready only after it reports runtime readiness. In poll mode, which stays the default, supervisors keep polling. The supervisor must connect to every remote middleware required by the initial effective policy before starting the workload, even when fail_open is configured. A failed live registry reload retains the last-known-good registry and keeps the workload running.

Stage 2 architecture walkthrough

Related Issue

Part of #1731.

Stacked on #3244.

Changes

• Negotiate streamed apply with supports_config_apply and config_apply_enabled, opt-in through push mode.
• Make the stream authoritative for bootstrap and live configuration, including startup policy preparation, for sessions that enable apply.
• Deliver through the Stage 1 delivery queue, holding each component until the supervisor acknowledges the previous update.
• Resume polling when a session does not enable apply, for example after a rollback to poll mode.
• Record sanitized application outcomes and preserve policy fallback behavior.

Testing

• mise run pre-commit; openshell-server, openshell-supervisor, and openshell-supervisor-process tests; mise run go:ci; mise run sdk:ts:ci
• mise run ci and mise run e2e:docker have not been rerun since the port onto the Stage 1 delivery queue.

Known reliability issue observed under load

A focused reproduction with 32 running sandboxes and 32 concurrent exec clients produced 6 false exit-code-1 results across 37,376 exec calls, including commands that explicitly ran exit 0. Diagnostic logs confirmed that the orphan reaper collected a child's exit status 0 before SSH's waiter received ECHILD and substituted exit code 1. No policy changes or gateway restart were needed.

The implicated spawn/register/reap code predates this PR. Existing fix commit c5cf4ec2b is included in #3142.

Follow-ups

Retry failed builds with backoff, record a durable fanout watermark, reconcile owner handoffs, and roll back builds whose session owner is briefly stale. The 30-second owner reconciler covers missed deliveries until then.

Checklist

• Conventional commits with DCO sign-off
• Published docs unchanged: streamed apply is internal to the push rollout
• Related public skill reviewed and updated
• No generated TypeScript sources committed

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3265.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[pimlock]

Added the startup policy preparation exchange in fa58bddb8.

Before this change, the supervisor discovered or enriched the startup policy, called the unary UpdateConfig RPC, then reconnected so it could receive a fresh authoritative bootstrap. That extra mutation and reconnect made the startup ownership model harder to follow.

The initial ConnectSupervisor stream now carries the complete exchange:

1. SupervisorHello may offer the policy found in the sandbox image.
2. The gateway selects its existing policy when present. Otherwise it selects the image policy.
3. The gateway sends the selected full policy as StartupConfigCandidate.
4. The supervisor performs image-specific filesystem enrichment and replies with StartupConfigPrepared, using unchanged, a complete prepared policy, or a bounded failure.
5. The gateway treats a prepared policy as an untrusted proposal. It runs it through the normal sandbox policy update path, including validation, safety checks, persistence, and composition.
6. The gateway rebuilds the bootstrap and sends SessionAccepted. The supervisor initializes only from that bootstrap.

There is no reconnect after preparation. Later reconnects omit the image policy, skip startup preparation, and receive the current gateway bootstrap directly.

This keeps the gateway authoritative without requiring it to know what paths exist in the sandbox image. It also removes the old ambiguity around mutation responses: neither the candidate nor the preparation response becomes runtime state. SessionAccepted remains the single startup source of truth.

Failure behavior is fail-closed. Candidate mismatches, preparation failures, and invalid prepared policies produce SessionRejected before the supervisor can observe SessionAccepted. Session traffic is buffered until the remaining fallible acceptance work completes, so a config update cannot overtake acceptance.

The change includes protocol bindings, architecture documentation with a sequence diagram, positive tests for policy precedence and persistence, a negative invalid-policy test, and the focused Docker live-policy E2E coverage.

[pimlock]

gator-agent

PR Review Status

Thanks @pimlock. I reviewed the current stacked-base merge in critical-only mode against the durable feedback ledger. The merge resolution introduces no new Critical defect, and GATOR-c10f3724-01 through GATOR-c10f3724-04 remain resolved.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None; all four prior Gator findings remain resolved.

Gator metadata

• Validation: Project-valid stage 2 of accepted issue Push gateway-owned desired state to supervisors #1731, stacked on active stage-1 PR feat(supervisor): stage gateway configuration snapshot delivery #3244.
• Docs: The reviewed merge resolution does not introduce a new direct UX change; the PR's existing operator documentation remains present.
• Checks: Current-head Branch Checks and E2E are queued or running; completed required gates are green.
• E2E: test:e2e and test:e2e-kubernetes remain applied, and current-head workflows were dispatched without a rerun or /ok to test.
• Head SHA: 1dc9c881ea2b5ac75a93e6dca5e6ee65e1d0c165
• Base SHA: 2091c38a87979c98db51a73722e4e2a204906575
• Merge base SHA: 2091c38a87979c98db51a73722e4e2a204906575
• Patch ID: 8246581dd660c77df41e64fc0013fc444c069255
• Gator payload: 9
• Review mode: critical_only
• Previous reviewed SHA: bd0ad5ee590142391b8eda9cdbf9ed59ee772638
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[politerealism]

Hey @sjenning @mrunalp @derekwaynecarr — this one's been open a bit and is part of the #1731 staged rollout (this is stage 2 of 3). Any chance you have bandwidth for a review pass in the near term? Happy to help unblock if there's anything I can clarify in the meantime.

[pimlock]

Superseded by #4320, which combines #3244, #3265 and #3273 into one PR so the work can iterate in one place. Review history stays here.