Skip to content

Commit 02e80eb

Browse files
committed
feat(vm): move supervisor outside guest boundary
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
1 parent 5b589ab commit 02e80eb

54 files changed

Lines changed: 7953 additions & 10098 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -577,9 +577,32 @@ kubectl -n <sandbox-namespace> logs <sandbox-pod> -c openshell-supervisor-networ
577577
Use the VM driver logs and host diagnostics available in the user's environment. Verify:
578578

579579
- The VM driver process is running and reachable by the gateway.
580-
- The runtime rootfs exists and matches the expected architecture.
581-
- Host virtualization support is enabled.
582-
- The sandbox supervisor can establish its callback connection to the gateway.
580+
- The custom kernel and runtime rootfs exist and match the expected architecture.
581+
- Host virtualization support is enabled and the active process can read and write `/dev/kvm`.
582+
- The native host supervisor can establish its callback connection to the gateway.
583+
- The guest process leaf accepts the authenticated virtio-vsock control channel.
584+
585+
For the managed libkrun driver, inspect the per-sandbox host-supervisor and
586+
guest-console logs separately:
587+
588+
```bash
589+
rg -n 'vm|grpc_endpoint|guest_tls|state_dir' .cache/gateway-vm/gateway.toml
590+
stat /dev/kvm
591+
id
592+
find /tmp/openshell-vm-driver-*/sandboxes -name 'supervisor*.log' -o -name 'rootfs-console.log'
593+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/supervisor.err.log
594+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/rootfs-console.log
595+
```
596+
597+
The historical `guest_tls_*` configuration fields are host-supervisor mTLS
598+
paths and must never appear in the guest image. `grpc_endpoint` is also
599+
host-reachable; loopback is valid. A successful guest boot logs
600+
`VM process supervisor leaf listening on vsock port ...`. libkrun needs KVM but
601+
does not need `CAP_NET_ADMIN`. QEMU/VFIO uses TAP and host nftables and still
602+
requires the corresponding networking and device privileges. If the account is
603+
listed in the group that owns `/dev/kvm` but `id` does not show that group, use
604+
`mise run gateway:vm` or the VM e2e runner; they re-exec through the configured
605+
group without sudo.
583606

584607
Then run:
585608

‎.github/workflows/driver-vm-linux.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ jobs:
116116
username: ${{ github.actor }}
117117
password: ${{ secrets.GITHUB_TOKEN }}
118118
options: --privileged
119+
volumes:
120+
- /var/run/docker.sock:/var/run/docker.sock
119121
env:
120122
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
121123
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -141,8 +143,12 @@ jobs:
141143
cache-directories: .cache/sccache
142144
cache-targets: "true"
143145

144-
- name: Install zstd
145-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
146+
- name: Install zstd and verify Docker
147+
run: |
148+
apt-get update
149+
apt-get install -y --no-install-recommends zstd
150+
rm -rf /var/lib/apt/lists/*
151+
docker info
146152
147153
- name: Download kernel runtime tarball
148154
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -168,15 +174,10 @@ jobs:
168174
- name: Verify embedded driver inputs
169175
run: |
170176
set -euo pipefail
171-
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
177+
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
172178
test -s "target/vm-runtime-compressed/${file}"
173179
done
174180
175-
- name: Scope workspace to driver-vm crates
176-
run: |
177-
set -euo pipefail
178-
sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core"]|' Cargo.toml
179-
180181
- name: Patch workspace version
181182
if: ${{ inputs['cargo-version'] != '' }}
182183
run: |

‎.github/workflows/driver-vm-macos.yml‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ jobs:
7575
credentials:
7676
username: ${{ github.actor }}
7777
password: ${{ secrets.GITHUB_TOKEN }}
78+
volumes:
79+
- /var/run/docker.sock:/var/run/docker.sock
7880
env:
7981
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
8082
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -100,8 +102,12 @@ jobs:
100102
cache-directories: .cache/sccache
101103
cache-targets: "true"
102104

103-
- name: Install zstd
104-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
105+
- name: Install zstd and verify Docker
106+
run: |
107+
apt-get update
108+
apt-get install -y --no-install-recommends zstd
109+
rm -rf /var/lib/apt/lists/*
110+
docker info
105111
106112
- name: Build bundled supervisor
107113
run: |
@@ -116,7 +122,9 @@ jobs:
116122
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
117123
with:
118124
name: driver-vm-supervisor-arm64
119-
path: target/vm-runtime-compressed/openshell-sandbox.zst
125+
path: |
126+
target/vm-runtime-compressed/openshell-sandbox.zst
127+
target/vm-runtime-compressed/openshell-runtime.tar.zst
120128
retention-days: 1
121129

122130
build-driver-vm-macos:
@@ -180,12 +188,13 @@ jobs:
180188
run: |
181189
set -euo pipefail
182190
test -f target/vm-runtime-compressed-macos/openshell-sandbox.zst
183-
ls -lh target/vm-runtime-compressed-macos/openshell-sandbox.zst
191+
test -f target/vm-runtime-compressed-macos/openshell-runtime.tar.zst
192+
ls -lh target/vm-runtime-compressed-macos/openshell-{sandbox,runtime.tar}.zst
184193
185194
- name: Verify embedded driver inputs
186195
run: |
187196
set -euo pipefail
188-
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
197+
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
189198
test -s "target/vm-runtime-compressed-macos/${file}"
190199
done
191200
@@ -203,14 +212,16 @@ jobs:
203212
.
204213
205214
- name: Verify packaged binary shape
206-
run: test -x out/openshell-driver-vm
215+
run: |
216+
test -x out/openshell-driver-vm
217+
test -x out/openshell-sandbox
207218
208219
- name: Package binary
209220
run: |
210221
set -euo pipefail
211222
mkdir -p artifacts
212223
tar -czf artifacts/openshell-driver-vm-aarch64-apple-darwin.tar.gz \
213-
-C out openshell-driver-vm
224+
-C out openshell-driver-vm openshell-sandbox
214225
ls -lh artifacts/
215226
216227
- name: Upload artifact

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
3333
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
3434
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
3535
| `crates/openshell-isolation/` | Isolation backend contract | RFC 0012 `IsolationBackend` trait + types; the supervisor-facing runtime contract for the boundary |
36+
| `crates/openshell-isolation-vm/` | VM isolation transport | Shared authenticated host/guest boundary transport and portable process leaf for VM drivers |
3637
| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints |
3738
| `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing |
3839
| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
@@ -51,7 +52,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
5152
| `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods |
5253
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
5354
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
54-
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
55+
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` with a native host supervisor and embedded guest runtime |
5556
| `crates/openshell-prover/` | Policy prover | Policy verification and proof generation |
5657
| `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization |
5758
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |

‎Cargo.lock‎

Lines changed: 25 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/build.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ OpenShell builds these main artifacts:
1717
| Gateway container image | `deploy/docker/Dockerfile.gateway` |
1818
| Supervisor container image | `deploy/docker/Dockerfile.supervisor` |
1919
| Helm chart | `deploy/helm/openshell` |
20-
| VM driver/runtime assets | `crates/openshell-driver-vm` |
20+
| VM driver/runtime assets | `crates/openshell-driver-vm` plus shared transport in `crates/openshell-isolation-vm` |
2121
| Published docs site | `docs/` rendered by Fern config in `fern/` |
2222

2323
Sandbox community images are built outside this repository.
@@ -173,6 +173,16 @@ Runtime layout:
173173
enforcement. The VM driver bundles its own supervisor build
174174
(`tasks/scripts/vm/build-supervisor-bundle.sh`) and does not read
175175
`SUPERVISOR_LIBC`.
176+
before publishing artifacts. On Linux, the driver can materialize the
177+
same-target embedded `openshell-sandbox` as its native host supervisor. The
178+
macOS VM-driver archive includes a native `openshell-sandbox` sibling because
179+
the embedded guest leaf is a Linux binary.
180+
- **Supervisor**: Alpine base with `nftables`, static musl binary at
181+
`/openshell-sandbox`. Static linkage keeps the binary usable when the image
182+
is mounted or extracted into sandbox environments. The VM bundle build also
183+
packages the Linux process leaf and a driver-controlled network-helper
184+
runtime into the guest bootstrap while the logical VM supervisor runs
185+
natively on the host.
176186

177187
Gateway image builds bake the corresponding supervisor image tag into the
178188
gateway binary so Docker sandboxes do not depend on `:latest` by default.

‎architecture/compute-runtimes.md‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ Compute runtimes create, stop, start, delete, and watch sandbox workloads for th
44
gateway. They do not replace sandbox policy enforcement. Every runtime starts a
55
workload that runs the `openshell-sandbox` supervisor, and the supervisor
66
enforces the sandbox contract locally.
7+
Compute runtimes create, stop, delete, and watch sandbox workloads for the
8+
gateway. They do not replace sandbox policy enforcement. Container runtimes run
9+
the logical `openshell-sandbox` supervisor in the workload boundary. VM runtimes
10+
may instead run it on the host and use an authenticated process leaf inside the
11+
guest. In both placements the logical supervisor owns policy and the gateway
12+
session.
713

814
## Driver Contract
915

@@ -16,6 +22,9 @@ Each runtime receives a sandbox spec from the gateway and is responsible for:
1622
- Forwarding the exact canonical main-process argv and TTY mode without shell
1723
reconstruction. The sandbox-level environment and policy workspace apply to
1824
the main process.
25+
- Injecting sandbox identity and the admitted topology descriptor.
26+
- Supplying TLS or secret material only to the logical supervisor placement.
27+
- Providing the logical supervisor and any boundary-local process leaf.
1928
- Reporting lifecycle and platform events back to the gateway.
2029
- Cleaning up runtime-owned resources.
2130

@@ -207,6 +216,8 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
207216
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
208217
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
209218
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
219+
| VM | Experimental host-supervised microVM isolation. | Per-sandbox libkrun VM with a portable guest process leaf. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`; the driver starts a native logical supervisor and connects it to the guest over authenticated virtio-vsock. The guest has no gateway credentials. The existing custom kernel, cached bootstrap `rootfs.ext4`, in-VM `umoci` preparation, read-only image disk, writable `overlay.ext4`, and restart persistence remain unchanged. |
220+
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
210221

211222
Per-sandbox CPU and memory values currently enter the driver layer through
212223
template resource limits. Docker and Podman apply them as runtime limits.
@@ -255,25 +266,29 @@ Runtime-specific implementation notes belong in the driver crate README:
255266
- `crates/openshell-driver-kubernetes/README.md`
256267
- `crates/openshell-driver-vm/README.md`
257268

258-
The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
259-
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
260-
that identity when it execs the supervisor after mounting and network setup.
269+
The VM topology is delegated. libkrun executes the driver-owned guest bootstrap
270+
as PID 1; after mounting and network setup it execs `openshell-sandbox vm-guest`
271+
as the portable process leaf. The native host supervisor drives RFC 0012 over a
272+
driver-private, token-authenticated virtio-vsock transport. Lifecycle,
273+
exec/PTY, loopback forwarding, and mediated egress all cross this transport;
274+
gateway JWT and mTLS material remain on the host.
261275

262276
## Supervisor Delivery
263277

264-
The supervisor must be available inside each sandbox workload:
278+
The logical supervisor or its boundary-local process leaf must be available at
279+
the placement selected by the runtime:
265280

266281
| Runtime | Delivery model |
267282
|---|---|
268283
| Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
269284
| Podman | Read-only OCI image volume by default; host-cached bind mount when `userns` is configured. |
270285
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
271-
| VM | Embedded in the guest rootfs bundle. |
286+
| VM | Native host `openshell-sandbox` beside the driver; portable Linux process leaf embedded in the guest rootfs bundle. |
272287
| Extension | Defined by the out-of-tree driver. |
273288

274289
Driver-controlled environment variables must override sandbox image or template
275-
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
276-
paths, and command metadata.
290+
values for sandbox ID, sandbox name, relay path, and command metadata. Gateway
291+
endpoint and TLS values go only to the logical supervisor placement.
277292

278293
## Process Identity
279294

@@ -285,7 +300,8 @@ driver then supplies one authoritative identity input to the supervisor:
285300
resolves the workspace from OCI `Config.WorkingDir` during that inspection.
286301
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
287302
SCC-derived values.
288-
- VM keeps its existing guest identity behavior.
303+
- VM resolves the configured numeric UID/GID on the host and transfers the pair
304+
to the guest leaf as authenticated launch state.
289305

290306
Explicit numeric workload identities may use any Linux UID/GID from `1`
291307
through `u32::MAX - 1`. UID/GID `0` remains prohibited as root, and

0 commit comments

Comments
 (0)