Skip to content

Commit 532fa1b

Browse files
committed
feat(vm): move supervisor outside guest boundary
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
1 parent fccbbc0 commit 532fa1b

36 files changed

Lines changed: 4253 additions & 772 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -411,9 +411,32 @@ kubectl -n <sandbox-namespace> logs <sandbox-pod> -c agent --tail=200
411411
Use the VM driver logs and host diagnostics available in the user's environment. Verify:
412412

413413
- The VM driver process is running and reachable by the gateway.
414-
- The runtime rootfs exists and matches the expected architecture.
415-
- Host virtualization support is enabled.
416-
- The sandbox supervisor can establish its callback connection to the gateway.
414+
- The custom kernel and runtime rootfs exist and match the expected architecture.
415+
- Host virtualization support is enabled and the active process can read and write `/dev/kvm`.
416+
- The native host supervisor can establish its callback connection to the gateway.
417+
- The guest process leaf accepts the authenticated virtio-vsock control channel.
418+
419+
For the managed libkrun driver, inspect the per-sandbox host-supervisor and
420+
guest-console logs separately:
421+
422+
```bash
423+
rg -n 'vm|grpc_endpoint|guest_tls|state_dir' .cache/gateway-vm/gateway.toml
424+
stat /dev/kvm
425+
id
426+
find /tmp/openshell-vm-driver-*/sandboxes -name 'supervisor*.log' -o -name 'rootfs-console.log'
427+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/supervisor.err.log
428+
tail -n 200 /tmp/openshell-vm-driver-*/sandboxes/*/rootfs-console.log
429+
```
430+
431+
The historical `guest_tls_*` configuration fields are host-supervisor mTLS
432+
paths and must never appear in the guest image. `grpc_endpoint` is also
433+
host-reachable; loopback is valid. A successful guest boot logs
434+
`VM process supervisor leaf listening on vsock port ...`. libkrun needs KVM but
435+
does not need `CAP_NET_ADMIN`. QEMU/VFIO uses TAP and host nftables and still
436+
requires the corresponding networking and device privileges. If the account is
437+
listed in the group that owns `/dev/kvm` but `id` does not show that group, use
438+
`mise run gateway:vm` or the VM e2e runner; they re-exec through the configured
439+
group without sudo.
417440

418441
Then run:
419442

‎.github/workflows/driver-vm-linux.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ jobs:
116116
username: ${{ github.actor }}
117117
password: ${{ secrets.GITHUB_TOKEN }}
118118
options: --privileged
119+
volumes:
120+
- /var/run/docker.sock:/var/run/docker.sock
119121
env:
120122
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
121123
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -141,8 +143,12 @@ jobs:
141143
cache-directories: .cache/sccache
142144
cache-targets: "true"
143145

144-
- name: Install zstd
145-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
146+
- name: Install zstd and verify Docker
147+
run: |
148+
apt-get update
149+
apt-get install -y --no-install-recommends zstd
150+
rm -rf /var/lib/apt/lists/*
151+
docker info
146152
147153
- name: Download kernel runtime tarball
148154
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -168,15 +174,10 @@ jobs:
168174
- name: Verify embedded driver inputs
169175
run: |
170176
set -euo pipefail
171-
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
177+
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
172178
test -s "target/vm-runtime-compressed/${file}"
173179
done
174180
175-
- name: Scope workspace to driver-vm crates
176-
run: |
177-
set -euo pipefail
178-
sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core"]|' Cargo.toml
179-
180181
- name: Patch workspace version
181182
if: ${{ inputs['cargo-version'] != '' }}
182183
run: |

‎.github/workflows/driver-vm-macos.yml‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ jobs:
7575
credentials:
7676
username: ${{ github.actor }}
7777
password: ${{ secrets.GITHUB_TOKEN }}
78+
volumes:
79+
- /var/run/docker.sock:/var/run/docker.sock
7880
env:
7981
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
8082
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -100,8 +102,12 @@ jobs:
100102
cache-directories: .cache/sccache
101103
cache-targets: "true"
102104

103-
- name: Install zstd
104-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
105+
- name: Install zstd and verify Docker
106+
run: |
107+
apt-get update
108+
apt-get install -y --no-install-recommends zstd
109+
rm -rf /var/lib/apt/lists/*
110+
docker info
105111
106112
- name: Build bundled supervisor
107113
run: |
@@ -116,7 +122,9 @@ jobs:
116122
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
117123
with:
118124
name: driver-vm-supervisor-arm64
119-
path: target/vm-runtime-compressed/openshell-sandbox.zst
125+
path: |
126+
target/vm-runtime-compressed/openshell-sandbox.zst
127+
target/vm-runtime-compressed/openshell-runtime.tar.zst
120128
retention-days: 1
121129

122130
build-driver-vm-macos:
@@ -180,12 +188,13 @@ jobs:
180188
run: |
181189
set -euo pipefail
182190
test -f target/vm-runtime-compressed-macos/openshell-sandbox.zst
183-
ls -lh target/vm-runtime-compressed-macos/openshell-sandbox.zst
191+
test -f target/vm-runtime-compressed-macos/openshell-runtime.tar.zst
192+
ls -lh target/vm-runtime-compressed-macos/openshell-{sandbox,runtime.tar}.zst
184193
185194
- name: Verify embedded driver inputs
186195
run: |
187196
set -euo pipefail
188-
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
197+
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
189198
test -s "target/vm-runtime-compressed-macos/${file}"
190199
done
191200
@@ -203,14 +212,16 @@ jobs:
203212
.
204213
205214
- name: Verify packaged binary shape
206-
run: test -x out/openshell-driver-vm
215+
run: |
216+
test -x out/openshell-driver-vm
217+
test -x out/openshell-sandbox
207218
208219
- name: Package binary
209220
run: |
210221
set -euo pipefail
211222
mkdir -p artifacts
212223
tar -czf artifacts/openshell-driver-vm-aarch64-apple-darwin.tar.gz \
213-
-C out openshell-driver-vm
224+
-C out openshell-driver-vm openshell-sandbox
214225
ls -lh artifacts/
215226
216227
- name: Upload artifact

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
3333
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
3434
| `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing |
3535
| `crates/openshell-isolation/` | Isolation backend contract | RFC 0012 `IsolationBackend` trait + types; the supervisor-facing runtime contract for the boundary |
36+
| `crates/openshell-isolation-vm/` | VM isolation transport | Shared authenticated host/guest boundary transport and portable process leaf for VM drivers |
3637
| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints |
3738
| `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing |
3839
| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage |
@@ -46,7 +47,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
4647
| `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods |
4748
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
4849
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
49-
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
50+
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` with a native host supervisor and embedded guest runtime |
5051
| `crates/openshell-prover/` | Policy prover | Policy verification and proof generation |
5152
| `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization |
5253
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |

‎Cargo.lock‎

Lines changed: 26 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/build.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ OpenShell builds these main artifacts:
1515
| Gateway container image | `deploy/docker/Dockerfile.gateway` |
1616
| Supervisor container image | `deploy/docker/Dockerfile.supervisor` |
1717
| Helm chart | `deploy/helm/openshell` |
18-
| VM driver/runtime assets | `crates/openshell-driver-vm` |
18+
| VM driver/runtime assets | `crates/openshell-driver-vm` plus shared transport in `crates/openshell-isolation-vm` |
1919
| Published docs site | `docs/` rendered by Fern config in `fern/` |
2020

2121
Sandbox community images are built outside this repository.
@@ -95,12 +95,16 @@ Runtime layout:
9595
`/usr/libexec/openshell/openshell-driver-vm` in Linux packages and published
9696
as a release artifact. Linux GNU VM driver binaries must not reference
9797
`GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this
98-
before publishing artifacts.
98+
before publishing artifacts. On Linux, the driver can materialize the
99+
same-target embedded `openshell-sandbox` as its native host supervisor. The
100+
macOS VM-driver archive includes a native `openshell-sandbox` sibling because
101+
the embedded guest leaf is a Linux binary.
99102
- **Supervisor**: Alpine base with `nftables`, static musl binary at
100103
`/openshell-sandbox`. Static linkage keeps the binary usable when the image
101-
is mounted/extracted into sandbox environments (Docker extraction, Podman
102-
image volumes, Kubernetes init-container copy-self), while `nftables` supports
103-
Kubernetes supervisor sidecar egress enforcement.
104+
is mounted or extracted into sandbox environments. The VM bundle build also
105+
packages the Linux process leaf and a driver-controlled network-helper
106+
runtime into the guest bootstrap while the logical VM supervisor runs
107+
natively on the host.
104108

105109
Gateway image builds bake the corresponding supervisor image tag into the
106110
gateway binary so Docker sandboxes do not depend on `:latest` by default.

‎architecture/compute-runtimes.md‎

Lines changed: 22 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
11
# Compute Runtimes
22

33
Compute runtimes create, stop, delete, and watch sandbox workloads for the
4-
gateway. They do not replace sandbox policy enforcement. Every runtime starts a
5-
workload that runs the `openshell-sandbox` supervisor, and the supervisor
6-
enforces the sandbox contract locally.
4+
gateway. They do not replace sandbox policy enforcement. Container runtimes run
5+
the logical `openshell-sandbox` supervisor in the workload boundary. VM runtimes
6+
may instead run it on the host and use an authenticated process leaf inside the
7+
guest. In both placements the logical supervisor owns policy and the gateway
8+
session.
79

810
## Driver Contract
911

1012
Each runtime receives a sandbox spec from the gateway and is responsible for:
1113

1214
- Selecting the sandbox image.
13-
- Injecting sandbox identity and gateway callback configuration.
14-
- Supplying TLS or secret material for supervisor callbacks.
15-
- Providing the supervisor binary or image in the workload.
15+
- Injecting sandbox identity and the admitted topology descriptor.
16+
- Supplying TLS or secret material only to the logical supervisor placement.
17+
- Providing the logical supervisor and any boundary-local process leaf.
1618
- Reporting lifecycle and platform events back to the gateway.
1719
- Cleaning up runtime-owned resources.
1820

@@ -119,7 +121,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
119121
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
120122
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
121123
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
122-
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
124+
| VM | Experimental host-supervised microVM isolation. | Per-sandbox libkrun VM with a portable guest process leaf. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`; the driver starts a native logical supervisor and connects it to the guest over authenticated virtio-vsock. The guest has no gateway credentials. The existing custom kernel, cached bootstrap `rootfs.ext4`, in-VM `umoci` preparation, read-only image disk, writable `overlay.ext4`, and restart persistence remain unchanged. |
123125
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
124126

125127
Per-sandbox CPU and memory values currently enter the driver layer through
@@ -159,25 +161,29 @@ Runtime-specific implementation notes belong in the driver crate README:
159161
- `crates/openshell-driver-kubernetes/README.md`
160162
- `crates/openshell-driver-vm/README.md`
161163

162-
The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
163-
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
164-
that identity when it execs the supervisor after mounting and network setup.
164+
The VM topology is delegated. libkrun executes the driver-owned guest bootstrap
165+
as PID 1; after mounting and network setup it execs `openshell-sandbox vm-guest`
166+
as the portable process leaf. The native host supervisor drives RFC 0012 over a
167+
driver-private, token-authenticated virtio-vsock transport. Lifecycle,
168+
exec/PTY, loopback forwarding, and mediated egress all cross this transport;
169+
gateway JWT and mTLS material remain on the host.
165170

166171
## Supervisor Delivery
167172

168-
The supervisor must be available inside each sandbox workload:
173+
The logical supervisor or its boundary-local process leaf must be available at
174+
the placement selected by the runtime:
169175

170176
| Runtime | Delivery model |
171177
|---|---|
172178
| Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
173179
| Podman | Read-only OCI image volume containing the supervisor binary. |
174180
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
175-
| VM | Embedded in the guest rootfs bundle. |
181+
| VM | Native host `openshell-sandbox` beside the driver; portable Linux process leaf embedded in the guest rootfs bundle. |
176182
| Extension | Defined by the out-of-tree driver. |
177183

178184
Driver-controlled environment variables must override sandbox image or template
179-
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
180-
paths, and command metadata.
185+
values for sandbox ID, sandbox name, relay path, and command metadata. Gateway
186+
endpoint and TLS values go only to the logical supervisor placement.
181187

182188
## Process Identity
183189

@@ -188,7 +194,8 @@ driver then supplies one authoritative identity input to the supervisor:
188194
its immutable image ID, and pass its raw OCI `Config.User`.
189195
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
190196
SCC-derived values.
191-
- VM keeps its existing guest identity behavior.
197+
- VM resolves the configured numeric UID/GID on the host and transfers the pair
198+
to the guest leaf as authenticated launch state.
192199

193200
For Docker and Podman, policy values take precedence independently. An omitted
194201
`run_as_user` or `run_as_group` falls back to the corresponding identity from

‎architecture/sandbox.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,36 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
4949
The supervisor may enrich baseline filesystem allowances for runtime-required
5050
paths, such as proxy support files or GPU device paths when a GPU is present.
5151

52+
## Isolation Backend
53+
54+
[RFC 0012](../rfc/0012-isolation-backend/README.md) defines the Isolation
55+
Backend contract for topology-specific boundary construction and process
56+
operations. The contract uses consuming
57+
lifecycle states (`attach` → `Bound` → `confirm` → `Ready` → `start_agent` →
58+
`Running`) so untrusted workload execution cannot begin before standing
59+
enforcement is confirmed.
60+
61+
The logical supervisor remains the trusted bridge between the gateway and the
62+
workload. It drives the backend and applies approved network policy through
63+
supervisor-owned mediation; the backend routes workload egress to that
64+
mediation. Existing container placements remain on their legacy lifecycle
65+
while the VM driver prototypes the backend contract.
66+
67+
VM uses delegated placement. The logical supervisor stays on
68+
the host and sends admitted policy, workload state, and proxy CA material over
69+
a token-authenticated, backend-private virtio-vsock channel only after the RFC
70+
lifecycle reaches `start_agent`. The driver uses the portable
71+
`openshell-isolation-vm` guest leaf, which invokes the existing
72+
`openshell-supervisor-process` implementation inside the VM. The guest resolves
73+
workload socket/process identity from its own `/proc` and sends that evidence
74+
with network connections; the host supervisor performs policy evaluation and
75+
relays approved traffic. Exec/PTY and loopback forwarding use the same boundary
76+
transport. Gateway JWT and mTLS credentials never enter the guest.
77+
78+
The VM guest installs and verifies its default-deny kernel egress ceiling before
79+
the host exposes workload execution. Boundary failure leaves the ceiling in
80+
place and triggers guest process cleanup.
81+
5282
## Network and Inference
5383

5484
All ordinary agent egress is routed through the sandbox proxy. The proxy

0 commit comments

Comments
 (0)