Skip to content

Commit 4a8f295

Browse files
committed
feat(vm): move supervisor outside guest boundary
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
1 parent 531ffd5 commit 4a8f295

33 files changed

Lines changed: 3722 additions & 807 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -425,9 +425,32 @@ kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200
425425
Use the VM driver logs and host diagnostics available in the user's environment. Verify:
426426

427427
- The VM driver process is running and reachable by the gateway.
428-
- The runtime rootfs exists and matches the expected architecture.
429-
- Host virtualization support is enabled.
430-
- The sandbox supervisor can establish its callback connection to the gateway.
428+
- The custom kernel and runtime rootfs exist and match the expected architecture.
429+
- Host virtualization support is enabled and the active process can read and write `/dev/kvm`.
430+
- The native host supervisor can establish its callback connection to the gateway.
431+
- The guest process leaf accepts the authenticated virtio-vsock control channel.
432+
433+
For the managed libkrun driver, inspect the per-sandbox host-supervisor and
434+
guest-console logs separately:
435+
436+
```bash
437+
rg -n 'vm|grpc_endpoint|guest_tls|state_dir' .cache/gateway-vm/gateway.toml
438+
stat /dev/kvm
439+
id
440+
find /tmp/openshell-vm-driver-*/sandboxes -name 'supervisor*.log' -o -name 'rootfs-console.log'
441+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/supervisor.err.log
442+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/rootfs-console.log
443+
```
444+
445+
The historical `guest_tls_*` configuration fields are host-supervisor mTLS
446+
paths and must never appear in the guest image. `grpc_endpoint` is also
447+
host-reachable; loopback is valid. A successful guest boot logs
448+
`VM process supervisor leaf listening on vsock port ...`. libkrun needs KVM but
449+
does not need `CAP_NET_ADMIN`. QEMU/VFIO uses TAP and host nftables and still
450+
requires the corresponding networking and device privileges. If the account is
451+
listed in the group that owns `/dev/kvm` but `id` does not show that group, use
452+
`mise run gateway:vm` or the VM e2e runner; they re-exec through the configured
453+
group without sudo.
431454

432455
Then run:
433456

‎.github/workflows/driver-vm-linux.yml‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -178,11 +178,6 @@ jobs:
178178
test -s "target/vm-runtime-compressed/${file}"
179179
done
180180
181-
- name: Scope workspace to driver-vm crates
182-
run: |
183-
set -euo pipefail
184-
sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core", "crates/openshell-isolation"]|' Cargo.toml
185-
186181
- name: Patch workspace version
187182
if: ${{ inputs['cargo-version'] != '' }}
188183
run: |

‎.github/workflows/driver-vm-macos.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -212,14 +212,16 @@ jobs:
212212
.
213213
214214
- name: Verify packaged binary shape
215-
run: test -x out/openshell-driver-vm
215+
run: |
216+
test -x out/openshell-driver-vm
217+
test -x out/openshell-sandbox
216218
217219
- name: Package binary
218220
run: |
219221
set -euo pipefail
220222
mkdir -p artifacts
221223
tar -czf artifacts/openshell-driver-vm-aarch64-apple-darwin.tar.gz \
222-
-C out openshell-driver-vm
224+
-C out openshell-driver-vm openshell-sandbox
223225
ls -lh artifacts/
224226
225227
- name: Upload artifact

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
3333
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
3434
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
3535
| `crates/openshell-isolation/` | Isolation backend contract | RFC 0012 `IsolationBackend` trait + types; the supervisor-facing runtime contract for the boundary |
36+
| `crates/openshell-isolation-vm/` | VM isolation transport | Shared authenticated host/guest boundary transport and portable process leaf for VM drivers |
3637
| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints |
3738
| `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing |
3839
| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
@@ -46,7 +47,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
4647
| `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods |
4748
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
4849
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
49-
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
50+
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` with a native host supervisor and embedded guest runtime |
5051
| `crates/openshell-prover/` | Policy prover | Policy verification and proof generation |
5152
| `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization |
5253
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |

‎Cargo.lock‎

Lines changed: 24 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/build.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ OpenShell builds these main artifacts:
1515
| Gateway container image | `deploy/docker/Dockerfile.gateway` |
1616
| Supervisor container image | `deploy/docker/Dockerfile.supervisor` |
1717
| Helm chart | `deploy/helm/openshell` |
18-
| VM driver/runtime assets | `crates/openshell-driver-vm` |
18+
| VM driver/runtime assets | `crates/openshell-driver-vm` plus shared transport in `crates/openshell-isolation-vm` |
1919
| Published docs site | `docs/` rendered by Fern config in `fern/` |
2020

2121
Sandbox community images are built outside this repository.
@@ -95,14 +95,18 @@ Runtime layout:
9595
`/usr/libexec/openshell/openshell-driver-vm` in Linux packages and published
9696
as a release artifact. Linux GNU VM driver binaries must not reference
9797
`GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this
98-
before publishing artifacts.
98+
before publishing artifacts. On Linux, the driver can materialize the
99+
same-target embedded `openshell-sandbox` as its native host supervisor. The
100+
macOS VM-driver archive includes a native `openshell-sandbox` sibling because
101+
the embedded guest leaf is a Linux binary.
99102
- **Supervisor**: Alpine base with a static musl binary at
100103
`/openshell-sandbox` and a materialized, driver-controlled network-helper
101104
runtime at `/openshell-runtime`. Kubernetes mounts or copies both into
102105
workload containers so boundary setup never executes helpers supplied by a
103106
workload image. Podman mounts the same image runtime, Docker extracts it to
104-
a digest-keyed read-only bind mount, and the VM supervisor build embeds it
105-
beside the guest supervisor.
107+
a digest-keyed read-only bind mount. The VM guest-leaf build embeds the Linux
108+
process leaf and helper runtime in the driver-controlled bootstrap while the
109+
logical VM supervisor runs natively on the host.
106110

107111
Gateway image builds bake the corresponding supervisor image tag into the
108112
gateway binary so Docker sandboxes do not depend on `:latest` by default.

‎architecture/compute-runtimes.md‎

Lines changed: 22 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
11
# Compute Runtimes
22

33
Compute runtimes create, stop, delete, and watch sandbox workloads for the
4-
gateway. They do not replace sandbox policy enforcement. Every runtime starts a
5-
workload that runs the `openshell-sandbox` supervisor, and the supervisor
6-
enforces the sandbox contract locally.
4+
gateway. They do not replace sandbox policy enforcement. Container runtimes run
5+
the logical `openshell-sandbox` supervisor in the workload boundary. VM runtimes
6+
may instead run it on the host and use an authenticated process leaf inside the
7+
guest. In both placements the logical supervisor owns policy and the gateway
8+
session.
79

810
## Driver Contract
911

1012
Each runtime receives a sandbox spec from the gateway and is responsible for:
1113

1214
- Selecting the sandbox image.
13-
- Injecting sandbox identity and gateway callback configuration.
14-
- Supplying TLS or secret material for supervisor callbacks.
15-
- Providing the supervisor binary or image in the workload.
15+
- Injecting sandbox identity and the admitted topology descriptor.
16+
- Supplying TLS or secret material only to the logical supervisor placement.
17+
- Providing the logical supervisor and any boundary-local process leaf.
1618
- Reporting lifecycle and platform events back to the gateway.
1719
- Cleaning up runtime-owned resources.
1820

@@ -119,7 +121,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
119121
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
120122
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
121123
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
122-
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
124+
| VM | Experimental host-supervised microVM isolation. | Per-sandbox libkrun VM with a portable guest process leaf. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`; the driver starts a native logical supervisor and connects it to the guest over authenticated virtio-vsock. The guest has no gateway credentials. The existing custom kernel, cached bootstrap `rootfs.ext4`, in-VM `umoci` preparation, read-only image disk, writable `overlay.ext4`, and restart persistence remain unchanged. |
123125
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
124126

125127
Per-sandbox CPU and memory values currently enter the driver layer through
@@ -159,25 +161,29 @@ Runtime-specific implementation notes belong in the driver crate README:
159161
- `crates/openshell-driver-kubernetes/README.md`
160162
- `crates/openshell-driver-vm/README.md`
161163

162-
The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
163-
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
164-
that identity when it execs the supervisor after mounting and network setup.
164+
The VM topology is delegated. libkrun executes the driver-owned guest bootstrap
165+
as PID 1; after mounting and network setup it execs `openshell-sandbox vm-guest`
166+
as the portable process leaf. The native host supervisor drives RFC 0012 over a
167+
driver-private, token-authenticated virtio-vsock transport. Lifecycle,
168+
exec/PTY, loopback forwarding, and mediated egress all cross this transport;
169+
gateway JWT and mTLS material remain on the host.
165170

166171
## Supervisor Delivery
167172

168-
The supervisor must be available inside each sandbox workload:
173+
The logical supervisor or its boundary-local process leaf must be available at
174+
the placement selected by the runtime:
169175

170176
| Runtime | Delivery model |
171177
|---|---|
172178
| Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
173179
| Podman | Read-only OCI image volume containing the supervisor binary. |
174180
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
175-
| VM | Embedded in the guest rootfs bundle. |
181+
| VM | Native host `openshell-sandbox` beside the driver; portable Linux process leaf embedded in the guest rootfs bundle. |
176182
| Extension | Defined by the out-of-tree driver. |
177183

178184
Driver-controlled environment variables must override sandbox image or template
179-
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
180-
paths, and command metadata.
185+
values for sandbox ID, sandbox name, relay path, and command metadata. Gateway
186+
endpoint and TLS values go only to the logical supervisor placement.
181187

182188
## Process Identity
183189

@@ -188,7 +194,8 @@ driver then supplies one authoritative identity input to the supervisor:
188194
its immutable image ID, and pass its raw OCI `Config.User`.
189195
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
190196
SCC-derived values.
191-
- VM keeps its existing guest identity behavior.
197+
- VM resolves the configured numeric UID/GID on the host and transfers the pair
198+
to the guest leaf as authenticated launch state.
192199

193200
For Docker and Podman, policy values take precedence independently. An omitted
194201
`run_as_user` or `run_as_group` falls back to the corresponding identity from

‎architecture/sandbox.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,22 @@ supervisor-owned mediation; the backend routes workload egress to that
6464
mediation. The trusted Kubernetes driver selects the co-located backend for
6565
combined topology and supplies its topology descriptor to the supervisor.
6666
Sidecar topology remains on its pre-RFC lifecycle; a conforming backend for
67-
that placement requires separate design and implementation. Docker, Podman,
68-
and VM drivers provision the same co-located topology and supply its descriptor
69-
by default. The co-located backend requires the
67+
that placement requires separate design and implementation. Docker and Podman
68+
also provision the co-located topology. The co-located backend requires the
7069
supervisor to own the execution environment's PID namespace so boundary
7170
teardown can terminate every remaining workload process.
7271

72+
VM uses delegated placement. The logical supervisor stays on
73+
the host and sends admitted policy, workload state, and proxy CA material over
74+
a token-authenticated, backend-private virtio-vsock channel only after the RFC
75+
lifecycle reaches `start_agent`. The driver uses the portable
76+
`openshell-isolation-vm` guest leaf, which invokes the existing
77+
`openshell-supervisor-process` implementation inside the VM. The guest resolves
78+
workload socket/process identity from its own `/proc` and sends that evidence
79+
with network connections; the host supervisor performs policy evaluation and
80+
relays approved traffic. Exec/PTY and loopback forwarding use the same boundary
81+
transport. Gateway JWT and mTLS credentials never enter the guest.
82+
7383
For proxy-mode boundaries, the co-located backend verifies its default-deny
7484
kernel egress ceiling before exposing any workload execution surface and then
7585
rechecks it every 250 milliseconds. Each check has a two-second deadline.

‎crates/openshell-driver-vm/Cargo.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,11 @@ path = "src/main.rs"
2121
[dependencies]
2222
openshell-core = { path = "../openshell-core", default-features = false }
2323
openshell-isolation = { path = "../openshell-isolation" }
24+
openshell-isolation-vm = { path = "../openshell-isolation-vm" }
2425
openshell-policy = { path = "../openshell-policy" }
2526
openshell-vfio = { path = "../openshell-vfio" }
2627

28+
base64 = { workspace = true }
2729
bollard = { version = "0.20", features = ["ssh"] }
2830
tokio = { workspace = true }
2931
tonic = { workspace = true, features = ["transport"] }
@@ -36,6 +38,7 @@ clap = { workspace = true }
3638
tracing = { workspace = true }
3739
tracing-subscriber = { workspace = true }
3840
miette = { workspace = true }
41+
rand = { workspace = true }
3942
url = { workspace = true }
4043
serde = { workspace = true }
4144
serde_json = { workspace = true }
@@ -57,6 +60,7 @@ telemetry = ["openshell-core/telemetry"]
5760

5861
[dev-dependencies]
5962
temp-env = "0.3"
63+
tempfile = "3"
6064

6165
# smol-rs/polling drives the BSD/macOS parent-death detection in
6266
# procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use

0 commit comments

Comments
 (0)