feat(server): place supervisor sessions by consistent hash

[FrostGod]

Summary

Spreads supervisor sessions across gateway replicas and sends a sandbox's long-lived connections (SSH, port forwards, exec) straight to the replica holding its session, instead of relaying through a peer.

Related Issue

Part of #3528: covers graceful redistribution of supervisor sessions (placement plus handoff on shutdown) and adds owner routing for long-lived connections. Capacity metrics, autoscaling, advisory-lock scoping, and watch batching are out of scope. Builds on #1868.

Changes

• Placement: replicas register in a shared membership table (expired rows are swept); a consistent hash picks each sandbox's replica. A supervisor on the wrong replica is redirected once and falls back to its original endpoint if that fails. Only supervisors that advertise supports_session_redirect are redirected, so older supervisors keep working across a gateway upgrade. Redirect dials verify TLS against the gateway Service name.
• Graceful handoff: a replica shutting down leaves the ring first, then redirects its sessions to their new owners instead of dropping them.
• Owner hint: the CLI asks GetSandbox for the owner (x-openshell-want-owner) and gets it back as x-openshell-owner; other callers pay no extra lookup. The CLI sends it as x-openshell-replica on SSH, forward, and exec, and retries once unrouted only when the request never reached a gateway.
• Helm: opt-in grpcRoute.replicaRouting.enabled adds one Service per replica and header-matched GRPCRoute rules. Requires workload.kind=statefulset, workload.allowMultiReplicaStatefulSet=true, and at most 15 replicas. Off by default; rendered output is unchanged when off.

Testing

• Unit tests and clippy pass for server, CLI, core, supervisor, and the Kubernetes driver; mise run helm:test, go:proto:check, and mise run pre-commit pass.
• kubernetes_ha_rebalancing e2e passes on a 2-node cluster (scale 2→3→2, pod rolls, 32 MiB sync during rolls).
• Routing, 42 checks across 6 sandboxes: peer relays dropped from 37 to 0, all checks pass. Still 42/42 with one replica's route broken (unrouted fallback).
• Gateway SIGKILL: new and old CLI recover the same way; no regression.

Checklist

• Conventional commits, signed off
• Tests added
• Docs updated (docs/kubernetes/ingress.mdx, Helm README, debug-openshell-cluster skill)
• Go / TypeScript SDK clients do not request the owner hint yet (follow-up)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[FrostGod]

/ok

[EmilienM]

Thanks for taking this on. Placing sessions where the ring wants them should cut out a lot of peer relay hops, and handing sessions off on shutdown is a nice touch. I read through it with a multi-replica Kubernetes deployment in mind and left comments inline.

The three I'd want sorted before merging: redirects failing TLS verification against pod addresses, supervisors built before this PR that can't decode SessionRedirect after a gateway upgrade, and the Go SDK bindings. The rest are smaller, and a few are nits.

[pimlock]

/ok to test 2bf8a00

[EmilienM]

I'd be comfortable approving but there are some follow ups to do (see comments).

[EmilienM]

LGTM

[drew]

/ok to test 25fd5ae

[pimlock]

/ok to test baa3fba

[FrostGod]

Flaky Tests, @drew , can you run /ok again