[Jetson][Sandbox] Sandbox GPU passthrough proof still fails for the non-root sandbox user on JetPack 6.2 IGX Orin on v0.0.131, including on the native OpenShell GPU path

Note

Re-verified reopening of #7610. That issue was closed on 2026-10-06 with no closing pull request. Its earlier close (2026-07-28, PR #7762) was reopened two days later, so a release-containment check reports #7762 as contained in v0.0.131 and looks green — it is not a fix. Re-verification on v0.0.131 reproduces the original symptom verbatim.

Description

On JetPack 6.2 (L4T R36.5.1) IGX Orin, GPU detection succeeds but the end-to-end sandbox GPU passthrough proof fails for the unprivileged in-sandbox user, so nemoclaw onboard aborts at the GPU proof gate.

This run did not use the Docker GPU compatibility path. It was run with NEMOCLAW_DOCKER_GPU_PATCH=0, and the log confirms Direct sandbox GPU enabled; allowing OpenShell GPU policy enrichment. The accepted direction on #8910 is that OpenShell native CDI becomes the single authority for Jetson device injection, mounts and supplemental groups. The symptom reproduces on that path, against the OpenShell build v0.0.131 ships.

Platform scope: reproduced on JetPack 6.2 (R36.5.1) IGX Orin, the same platform as the original report. Other Jetson variants not tested in this pass.

Regression: No — this is the original defect persisting.

Environment

Device:        NVIDIA IGX Orin Developer Kit
OS:            Ubuntu 22.04 (JetPack 6.2, L4T R36.5.1)
Architecture:  aarch64
Kernel:        5.15.0-1020-nvidia-tegra-igx
Docker:        29.6.2
OpenShell CLI: 0.0.116
NemoClaw:      v0.0.131

Steps to Reproduce

NEMOCLAW_DOCKER_GPU_PATCH=0 nemoclaw onboard --agent openclaw \
  --name {sandbox} --fresh --non-interactive --yes

Watch step [6/8] Creating sandbox and the GPU proof sequence.

Expected Result

The sandbox GPU proof passes and onboarding completes with sandbox GPU enabled.

Actual Result

Detection and the first two proofs pass; the CUDA proof fails and aborts onboarding:

✓ GPU proof passed: nvidia-smi when available
✓ GPU proof passed: /proc/<pid>/task/<tid>/comm write
⚠ GPU proof inconclusive: cuInit(0) via libcuda.so.1
  NvRmMemInitNvmap failed with Permission denied 356: Memory Manager Not supported
  ****NvRmMemMgrInit failed**** error type: 196626 cuInit(0)=999
⚠ Sandbox CUDA proof failed: cuInit(0) via libcuda.so.1
GPU proof failed inside an executable sandbox.
  sandbox_phase=Ready

This is the original report's Actual Result verbatim — same two passing proofs, same NvRmMemInitNvmap / 196626 / cuInit(0)=999 signature.

Isolation — the boundary is the sandbox execution path, not Docker passthrough

Same host, same build, same image, same unprivileged identity, run directly under Docker:

docker run --runtime nvidia -u 998 --group-add 44 --group-add 995 ...
  cuInit(0) = 0        (CUDA_SUCCESS)

An unprivileged user in a plain Docker container initialises CUDA successfully on this machine. Only the sandbox execution path fails. This matches the distinction already recorded on #7610 — direct Docker succeeds while execution through OpenShell does not — and is why a change in the raw error number between runs should not be read as a fix or a new regression. The question is which execution path is being measured.

Workaround

NEMOCLAW_SANDBOX_GPU=0 (CPU-only) lets onboarding complete, as before.

Separate host conditions encountered, not part of this defect

Recorded because they blocked the GPU gate from being reached at all and may affect anyone re-running this:

A stale OpenShell forward registry blocks all onboarding. openshell forward list held two rows with STATUS dead for ports 18789 and 18790 — the first two dashboard-port candidates — left by sandboxes that no longer exist. Onboarding failed with Cannot allocate dashboard port: NemoClaw could not prove OpenShell forward ownership, which names neither the forward, the port, nor the remedy. Nothing was listening on those ports (ss and lsof both showed the range empty). openshell forward stop 18789 / 18790 cleared it and onboarding proceeded immediately. This looks like a product gap in its own right; I plan to file it separately once I can reproduce it deliberately.
The sandbox DNS preflight picks a resolver that does not answer. NemoClaw selects the first syntactically usable upstream from /run/systemd/resolve/resolv.conf; on this host that is the default-route gateway, which does not serve DNS, while the next entry in the same file does. The preflight correctly detects the dead resolver and fails closed, but does not fall through to the next nameserver.

QA request: isolate the failing boundary

Please compare CUDA initialization with and without NemoClaw on the same JetPack 6.2 (L4T R36.5.1) IGX Orin. Run the three paths below with the same OpenShell version and digest-qualified ARM64 sandbox image. Use separate, disposable sandbox names. Keep the GPU selection, sandbox policy, UID, and supplemental groups matched where the paths support them. Record any difference; an unmatched pass or failure does not isolate the owner.

Use this probe in each container or sandbox as the non-root workload user:

id
stat -c '%n %a %u:%g %F' /dev/nvmap
python3 -c 'import ctypes; lib = ctypes.CDLL("libcuda.so.1"); rc = lib.cuInit(0); print(f"cuInit(0)={rc}"); raise SystemExit(0 if rc == 0 else 1)'

Docker baseline: Repeat the direct docker run --runtime nvidia -u 998 --group-add 44 --group-add 995 ... case documented above, with the same image digest. Confirm whether cuInit(0)=0 still holds.
OpenShell without NemoClaw: Create a fresh GPU sandbox by calling openshell sandbox create directly, then run the probe through openshell sandbox exec --name <direct-name> --no-tty -- sh -lc '<probe>'. Use the same --from image digest, --gpu selection, --policy file, and --driver-config-json values that NemoClaw supplies in step 3. A direct-create shape is openshell sandbox create --name <direct-name> --from <image@sha256:...> --gpu --policy <policy-file> --detach -- sleep 3600; add the matching driver config and GPU count or selector if present. Do not invoke nemoclaw to create this sandbox. If the generated policy or other input cannot be reused, state what differs.
NemoClaw: Reproduce with NEMOCLAW_DOCKER_GPU_PATCH=0 nemoclaw onboard --agent openclaw --name <nemoclaw-name> --fresh --non-interactive --yes. Capture its GPU proof result and the effective OpenShell create inputs. Do not enable the Docker compatibility path for this comparison.

For each path, please provide the command exit status, id, /dev/nvmap permissions, cuInit(0) output and stderr, and whether nvidia-smi and the /proc/<pid>/task/<tid>/comm proof pass when available. Also provide the JetPack/L4T, NemoClaw, OpenShell, Docker, image digest, selected CDI device/spec, effective policy, and redacted create/exec logs. Note any device, mount, or group difference between the direct OpenShell and NemoClaw sandboxes. Remove only the disposable test sandboxes and report whether cleanup succeeded. Do not post credentials or unredacted environment dumps.

Interpretation: If direct OpenShell passes and NemoClaw fails with matched inputs, investigate NemoClaw's creation or proof integration. If both sandbox paths fail while Docker passes, investigate OpenShell's GPU device, group, or policy handling. If the inputs differ or any path fails before the CUDA probe, report that result as inconclusive with the first failing step. Please test the existing released versions first; a later test of OpenShell PR #4204 should be a separate comparison.

Ran the 3-path comparison you asked for, with matched inputs where possible. Results below isolate the failure to OpenShell's own GPU device/group handling, not NemoClaw's sandbox-creation or proof integration.

Host / versions

NVIDIA IGX Orin Developer Kit, aarch64
JetPack 6.2 (L4T R36.5.1, GCID 45201625), kernel 5.15.0-1020-nvidia-tegra-igx
Driver 540.5.0, CUDA 12.6
nemoclaw v0.0.132 (upgraded from v0.0.131 for this test via NEMOCLAW_INSTALL_TAG=v0.0.132, confirmed with nemoclaw --version)
openshell 0.0.116
Docker 29.6.2, nvidia-container-toolkit 1.20.0-rc.1, CSV mode
Image digest used for all three paths: ghcr.io/nvidia/nemoclaw/openclaw-sandbox@sha256:473643565f557517f4139b0cf6ac552160f555cfd8f6241ef221144c25e9ed4d (this is the exact digest NemoClaw's own onboard run selected/built for this session — I ran step 3 first and reused its resolved digest for steps 1 and 2 to guarantee parity)
GPU selection: --gpu (auto/default count) in both OpenShell paths; NemoClaw's preflight reported "Sandbox GPU: enabled (auto)" / selected_gpu_route=native
Sandbox policy for both OpenShell paths: identical — pulled verbatim from the NemoClaw-created sandbox via openshell policy get <name> --base -o json and passed back in via --policy for the direct-create path (had to use --base, not the full effective policy, because the full policy has a _provider_*-prefixed network-policy key that sandbox create --policy rejects as a reserved name; this prefix is provider-composition metadata added after creation, not part of what a caller can set at create time, so it doesn't affect the filesystem/process/landlock portions that matter for the GPU comparison)
--driver-config-json: not captured — no CLI surface (sandbox get, policy get) or log I could find exposes the literal value NemoClaw passed for this field. Indirect signal: the outer OpenShell-managed container runs under runtime=runc with no Devices/GroupAdd/CDI involvement at the Docker level in either path's docker inspect, which suggests NemoClaw isn't passing Kubernetes-style driver config on this Docker-backed Jetson host. Flagging this as an explicit gap rather than guessing.

Probe used (as non-root sandbox/workload user)

id
stat -c '%n %a %u:%g %F' /dev/nvmap
python3 -c 'import ctypes; lib = ctypes.CDLL("libcuda.so.1"); rc = lib.cuInit(0); print(f"cuInit(0)={rc}"); raise SystemExit(0 if rc == 0 else 1)'

1. Docker baseline

docker run --rm --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all \
  -u 998 --group-add 44 --group-add 995 \
  ghcr.io/nvidia/nemoclaw/openclaw-sandbox@sha256:473643565f557517f4139b0cf6ac552160f555cfd8f6241ef221144c25e9ed4d \
  sh -lc '<probe>'

Exit status: 0
id: uid=998(sandbox) gid=998(sandbox) groups=998(sandbox),44(video),109,995
/dev/nvmap: 660 998:44 character special file
cuInit(0)=0 — PASS
nvidia-smi: not present in this image (n/a for this probe)
Note: NVIDIA_VISIBLE_DEVICES/NVIDIA_DRIVER_CAPABILITIES had to be set explicitly — without them the nvidia-container-runtime CSV hook mounts nothing (/dev/nvmap absent, libcuda.so.1 absent) even with --runtime nvidia. Standard Docker/CSV behavior, not NemoClaw-specific.

2. OpenShell direct (no `nemoclaw` invoked)

openshell sandbox create --name ipt12806-direct \
  --from ghcr.io/nvidia/nemoclaw/openclaw-sandbox@sha256:473643565f557517f4139b0cf6ac552160f555cfd8f6241ef221144c25e9ed4d \
  --gpu --policy <base-policy-from-nemoclaw-sandbox.yaml> --detach -- sleep 3600

openshell sandbox exec --name ipt12806-direct --no-tty -- sh -s < probe.sh

Create exit status: 0, sandbox reached Ready
Exec exit status: 1
id: uid=998(sandbox) gid=998(sandbox) groups=998(sandbox) — no supplementary groups at all
/dev/nvmap: 660 0:44 character special file (owner root, group 44/video — but the process's groups list above doesn't include 44)

cuInit(0)=999 — FAIL

NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported
****NvRmMemMgrInit failed**** error type: 196626

nvidia-smi: not present in this image (n/a)

3. NemoClaw

NEMOCLAW_DOCKER_GPU_PATCH=0 nemoclaw onboard --agent openclaw --name ipt12806-nc --fresh --non-interactive --yes

Preflight: GPU detected ("Orin (nvgpu), 54790 MB"), "Sandbox GPU: enabled (auto)"
Sandbox reached Ready; NemoClaw's own built-in GPU proof:
- ✓ nvidia-smi when available — passed (binary not present in this image, so this check is a no-op pass)
- ✓ /proc/<pid>/task/<tid>/comm write — passed
- ✗ cuInit(0) via libcuda.so.1 — FAIL, onboarding aborted: Error: Sandbox post-create verification or finalization failed
Re-ran the exact same probe directly against the resulting sandbox via openshell sandbox exec --name ipt12806-nc --no-tty:
- id: uid=998(sandbox) gid=998(sandbox) groups=998(sandbox) — identical to path 2, no supplementary groups
- /dev/nvmap: 660 0:44 character special file — identical to path 2
- cuInit(0)=999 — identical error text to path 2:
```
NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported
****NvRmMemMgrInit failed**** error type: 196626
```

Interpretation

Per your guide: both sandbox paths fail identically while Docker passes, with matched image digest, GPU selector, and policy → this points at OpenShell's own GPU device/group handling, not NemoClaw's sandbox-creation or proof-integration code. NemoClaw is accurately surfacing a failure that originates one layer down.

Concrete break point observed: /dev/nvmap is group-owned by video (gid 44) in both OpenShell-created sandboxes, but the sandboxed workload process's supplementary groups are empty (groups=998 only) in both — it never gets video added. The matched Docker baseline only passes because I added --group-add 44 explicitly. The generated sandbox policy's filesystem_policy.read_write list also only contains the discrete-GPU node names (/dev/nvidiactl, /dev/nvidia0) — none of the Tegra-specific /dev/nvhost-* nodes or /dev/nvmap appear in the policy's device list at all, even though /dev/nvmap is present (with correct Tegra group ownership) in the outer OpenShell-managed container. The gap looks like it's between how OpenShell's --gpu auto-selection populates the outer container's devices (seems to get this mostly right for Tegra) and how it translates that into the policy/group membership actually handed to the sandboxed, non-root exec'd process (misses the supplementary group and the Tegra-specific device paths).

Not yet tested

Per your instructions I tested only existing released versions (nemoclaw v0.0.132, openshell 0.0.116) in this pass. OpenShell PR #4204 was intentionally not included here — happy to run that as a separate follow-up comparison if useful.

Cleanup

Both disposable sandboxes (ipt12806-direct, ipt12806-nc) and their containers were deleted (openshell sandbox delete) and confirmed gone from docker ps -a / openshell sandbox list. One cosmetic leftover: NemoClaw's local retained-sandbox-recovery.json still has a metadata-only entry for ipt12806-nc (no container/sandbox behind it) — this is the same "retained after sandbox creation failure" bookkeeping pattern this host already had for two unrelated pre-existing sandbox names from an earlier session, not something new my test introduced.

OpenShell GPU PR status and IGX test request (October 16)

OpenShell PR 4204 is open and still a draft. At commit 33350a881f55011286e7c3a68d8007e43e50e0bf, Branch Checks, core E2E, GPU E2E, and Helm Lint passed. The GPU jobs passed on Linux ARM64, Linux AMD64, and WSL AMD64. The PR has no approving review. Its compatibility advisory still identifies possible breaking changes for OpenShell maintainers to assess. These checks do not yet prove that cuInit(0) works for the non-root user on this JetPack 6.2 IGX Orin.

Evan confirmed that this IGX configuration is in scope and suggested a CDI-based Docker control. Please use the same IGX host, image digest, non-root UID, and CUDA probe from the earlier three-path result:

Capture nvidia-ctk cdi list, Docker's reported CDISpecDirs, and redacted entries from /var/run/cdi/nvidia.yaml for /dev/nvmap and additionalGids. Record the Docker and NVIDIA Container Toolkit versions. If the spec is elsewhere, record its path.
Run the Docker control with --device=nvidia.com/gpu=all -u 998. Omit --runtime nvidia, NVIDIA_* environment variables, and all --group-add flags. Record id, /dev/nvmap ownership and mode, cuInit(0), stderr, and exit status. This checks whether CDI and Docker supply the required groups without manual help.
If the CDI Docker control passes, repeat the probe in direct OpenShell using a build identified by PR commit 33350a8 (or the newer commit being tested). Keep the image digest, GPU request, and base policy matched. If direct OpenShell passes, repeat with NemoClaw's native route (NEMOCLAW_DOCKER_GPU_PATCH=0). Record the tested OpenShell commit and the same identity, device, and CUDA results for each path. Use separate disposable sandbox names and report cleanup.

If the CDI Docker control fails, report that result before attributing the failure to OpenShell. If Docker passes but direct OpenShell fails, send the comparison to OpenShell PR 4204. If direct OpenShell passes but NemoClaw fails, investigate NemoClaw's create inputs or proof integration. Until GPU access is verified on this host, NEMOCLAW_SANDBOX_GPU=0 remains the CPU-only onboarding workaround; it is not GPU qualification. Please redact credentials and environment data from any logs.

CDI Docker control passes; direct OpenShell at `33350a8` cannot start on this host (Landlock ABI v1)

Ran the requested sequence on the same IGX Orin as the earlier three-path result, with the same image digest (ghcr.io/nvidia/nemoclaw/openclaw-sandbox@sha256:473643565f557517f4139b0cf6ac552160f555cfd8f6241ef221144c25e9ed4d), non-root UID 998 and CUDA probe.

Versions. JetPack 6.2 (L4T R36.5.1), kernel 5.15.0-1020-nvidia-tegra-igx, aarch64. Docker 29.9.1, NVIDIA Container Toolkit 1.19.0. Note: the earlier three-path run used Docker 29.6.2 and toolkit 1.20.0-rc.1; Docker was reinstalled and the toolkit is a different version this time.

1. CDI inventory

nvidia-ctk cdi list: nvidia.com/gpu=0, nvidia.com/gpu=all
Docker CDISpecDirs: /etc/cdi, /run/cdi; the spec is /var/run/cdi/nvidia.yaml
/dev/nvmap entry in the spec (both devices): major 10, minor 123, fileMode: 432 (0660), gid: 44
I found no additionalGids entry anywhere in the spec (only per-device-node gid: 44)

2. CDI Docker control: PASS

docker run --rm --device=nvidia.com/gpu=all -u 998 <digest> sh /probe.sh

(no --runtime nvidia, no NVIDIA_* variables, no --group-add)

id: uid=998(sandbox) gid=998(sandbox) groups=998(sandbox)
/dev/nvmap: 660 998:44 character special file
cuInit(0)=0, exit status 0

So Docker plus the CDI spec alone let the non-root user initialize CUDA on this host. In this path /dev/nvmap is owned by the container UID, whereas in the earlier direct-OpenShell run it was 0:44 with no supplementary groups and cuInit(0)=999.

3. Direct OpenShell with the PR build: could not run

Used the CI artifacts for 33350a8 from the PR's Branch E2E run: openshell and openshell-gateway 0.1.5-dev.47+g33350a881 (arm64 deb, extracted without installing), plus the commit's supervisor and sandbox images. Standalone gateway with the Docker driver on its own port and state directory (schema v2 config, plaintext listener, JWT bundle from generate-certs), separate from the existing NemoClaw gateway.

openshell sandbox create --name {sandbox} --from <digest> --gpu --policy <base policy> --detach -- sleep 3600

The sandbox goes to Error and the supervisor container exits before it is ready:

ControlSupervisorStartFailed: Docker sandbox exited before supervisor became ready
supervisor log:
Error: x Landlock ABI probe
  `-> Landlock ABI v3 or newer (Linux 6.2+) is required, but Landlock is available (ABI v1)

The kernel lists landlock in /sys/kernel/security/lsm, but JetPack 6.2's 5.15 kernel only provides ABI v1.

This does not look specific to #4204: crates/openshell-sandbox/src/linux/landlock.rs is byte-identical at 33350a8 and at v0.1.4, and docs/about/support-matrix.mdx lists "Landlock ABI 3 or newer, introduced in Linux 6.2" as required. The file at v0.1.3 has no ABI check. I did not run the v0.1.4 binaries.

Consequences for this test: direct OpenShell never reached the CUDA probe, so the third path (NemoClaw with NEMOCLAW_DOCKER_GPU_PATCH=0 on this build) was not run either, and I cannot say whether #4204 fixes cuInit(0) for this host. NemoClaw currently pins OpenShell 0.0.116, which does not have this requirement.

Question

Since the IGX configuration was confirmed in scope for #4204, is a kernel without Landlock ABI v3 (JetPack 6.2, Linux 5.15) expected to be supported by this OpenShell line, or does validating this change on IGX require a newer kernel (or a different supervisor requirement)? Happy to re-run the direct-OpenShell and NemoClaw paths on whichever build or host you point me to.

All test sandboxes, the standalone gateway, its network, images and state directories were removed afterwards.

Navigation Menu

Repository navigation

[Jetson][Sandbox] Sandbox GPU passthrough proof still fails for the non-root sandbox user on JetPack 6.2 IGX Orin on v0.0.131, including on the native OpenShell GPU path #12806

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Isolation — the boundary is the sandbox execution path, not Docker passthrough

Workaround

Separate host conditions encountered, not part of this defect

QA request: isolate the failing boundary

Host / versions

Probe used (as non-root sandbox/workload user)

1. Docker baseline

2. OpenShell direct (no `nemoclaw` invoked)

3. NemoClaw

Interpretation

Not yet tested

Cleanup

OpenShell GPU PR status and IGX test request (October 16)

CDI Docker control passes; direct OpenShell at `33350a8` cannot start on this host (Landlock ABI v1)

1. CDI inventory

2. CDI Docker control: PASS

3. Direct OpenShell with the PR build: could not run

Question

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Jetson][Sandbox] Sandbox GPU passthrough proof still fails for the non-root sandbox user on JetPack 6.2 IGX Orin on v0.0.131, including on the native OpenShell GPU path #12806

Description

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Isolation — the boundary is the sandbox execution path, not Docker passthrough

Workaround

Separate host conditions encountered, not part of this defect

Activity

rsliter commented on Oct 9, 2026

QA request: isolate the failing boundary

wangericnv commented on Oct 12, 2026

Host / versions

Probe used (as non-root sandbox/workload user)

1. Docker baseline

2. OpenShell direct (no nemoclaw invoked)

3. NemoClaw

Interpretation

Not yet tested

Cleanup

rsliter commented on Oct 16, 2026

OpenShell GPU PR status and IGX test request (October 16)

wangericnv commented on Oct 19, 2026

CDI Docker control passes; direct OpenShell at 33350a8 cannot start on this host (Landlock ABI v1)

1. CDI inventory

2. CDI Docker control: PASS

3. Direct OpenShell with the PR build: could not run

Question

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. OpenShell direct (no `nemoclaw` invoked)

CDI Docker control passes; direct OpenShell at `33350a8` cannot start on this host (Landlock ABI v1)