test(tmachine): assert CDI scenario behavior in a Docker guest

Add a cdi feature suite that creates a sandbox for each CDI scenario
by selecting its device explicitly with --gpu 1 and the Docker
cdi_devices driver option, then checks what the workload observes or
that creation is rejected. The sandbox only reports facts; the test
makes every assertion, so a failure names the check that disagreed.
Each scenario is its own test.

The expectations record the behavior the sandbox has today, loaded
from the scenarios.json published by the fixtures role. A
OPENSHELL_CDI_CAPTURE mode prints observations instead of asserting,
to record that behavior for a new scenario.

The test playbook enables caller driver configuration in the installed
gateway's active TOML for the run and restores it afterwards. Build the
suite's nextest archive in tests/artifacts.nix, expose it as
build-artifacts-cdi-tests, and run the suite on the Docker environment
with the deb installer in the branch and release matrices.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

Author: elezar

.github/workflows/branch-e2e.yml

@@ -275,6 +275,7 @@ jobs:
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
       test-matrix: >-
         [
+          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
           {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},

.github/workflows/release-dev.yml

@@ -145,6 +145,7 @@ jobs:
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
       test-matrix: >-
         [
+          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
           {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},

.github/workflows/release-qualification.yml

@@ -105,6 +105,7 @@ jobs:
       integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
       test-matrix: >-
         [
+          {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
           {"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
           {"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},

TESTING.md

@@ -330,6 +330,77 @@ nix run .#tmachine -- test fedora-podman-rootless binaries e2e-podman
 nix run .#tmachine -- test fedora-podman-rootless binaries driver-podman
 ```
 
+The `cdi` feature suite tests device injection without GPU hardware in the
+`ubuntu-docker-rootful` environment. Build the candidate artifacts, then run:
+
+```shell
+nix run .#build-artifacts
+nix run .#tmachine -- test ubuntu-docker-rootful deb cdi
+```
+
+If candidate binaries, images, and packages are already built, rebuild just the
+CDI test archive with `nix run .#build-artifacts-cdi-tests`.
+
+The `binaries` installer also supports this suite. Docker environment setup
+creates the named scenario fixtures and a CDI spec that defines one device per
+scenario, selected as `tmachine.example/cdi-scenario=<scenario>`. Persistent
+spec sources live under `/var/lib/openshell-test-inputs/cdi/specs`; a
+systemd-tmpfiles configuration recreates them in `/var/run/cdi` on every VM
+boot. This preserves the fixtures across tmachine's setup, installation, and
+test disk layers and makes them available to every suite using the Docker
+environment. The tests need no GPU hardware.
+
+Each test creates a sandbox that selects one scenario explicitly with `--gpu 1`
+and the Docker `cdi_devices` driver option, then checks what the workload
+observes or that creation is rejected. Selection by count or default, and the
+`nvidia.com/gpu` inventory, are not covered here. Caller driver configuration
+is enabled temporarily in the installed gateway's active TOML. The test
+playbook restores the configuration and restarts the gateway even after test
+failure; the CDI fixtures remain part of the environment. Docker must support
+CDI. Branch and release feature-test matrices run this suite on Docker.
+
+| Scenario | Injected resource | Asserted today | Expected with PR #4204 |
+| --- | --- | --- | --- |
+| `mount-under-read-only-baseline` | File at `/usr/share/tmachine-cdi/baseline-read-only-file`, under a directory the sandbox already makes read-only | Read succeeds, write denied. | Unchanged. |
+| `read-only-file` | File at `/tmachine-cdi/read-only/file` | Read and write denied. | Read succeeds outside the baseline grants; write fails. |
+| `read-only-directory` | Directory at `/tmachine-cdi/read-only/directory` | Reading its marker and creating files denied. | Reading its marker succeeds; write fails. |
+| `library-parent` | Marker named `/tmachine-cdi/library/libfixture.so.1` | Read and write denied. | Resolver derives the library parent; the marker is not an actual shared library. |
+| `broad-library-parent` | Marker named `/libtmachinefixture.so.1` | Read and write denied. | Launch rejects expansion to `/`. |
+| `writable-file` | Writable file at `/tmachine-cdi/writable/file` | Read and write denied. | Launch rejects it even with exact or ancestor write grants. |
+| `writable-directory` | Writable directory at `/tmachine-cdi/writable/directory` | Write denied. | Launch rejects it even with exact or ancestor write grants. |
+| `omitted-access-file` | Bind without `ro` or `rw` | Read and write denied. | Effective writable access causes launch rejection. |
+| `device-node` | Null character device at `/dev/tmachine-cdi-null` | Non-root, write denied. | Non-root read/write succeeds; an exact authored read-only entry fails launch. |
+| `device-node-with-gid` | Group-restricted null device at `/dev/tmachine-cdi-group-null` | Non-root with the device's group, write denied. | The generated supplementary GID permits non-root access. |
+| `unix-socket` | Echo socket at `/tmachine-cdi/socket/echo.sock` | Sibling file in the host directory is not visible. | Socket communication succeeds without deriving parent filesystem grants. |
+
+Every scenario also supplies a spec-level environment marker and a read-only
+mount at `/usr/share/tmachine-cdi/common/spec-marker`, allowing checks of
+spec/device edit merging without depending on derived grants. The device group
+is allocated in the disposable guest. A systemd socket unit restores the echo
+endpoint after reboot; the existing tmpfiles configuration restores the
+scenario spec. Negative scenarios must be selected individually, not combined
+with positive cases.
+
+The role publishes each scenario's expectations as `scenarios.json` next to the
+fixtures. The `scenario_*` tests load them and assert the behavior recorded in
+`vars/scenarios.yaml`, which is the sandbox's behavior today. A change that
+alters a scenario's behavior, such as #4204 deriving grants from the spec,
+updates those values in the same diff. To record the current behavior, run the
+tests with `OPENSHELL_CDI_CAPTURE=1` set for the test process; they then print
+what each scenario observed instead of asserting.
+
+To add a scenario, add an entry to `cdi_scenarios` in
+`tests/ansible/roles/cdi_fixtures/vars/scenarios.yaml` with the host fixtures
+it needs, its CDI edits, and its `expect` (the checks on the workload, or a
+rejection), then list it in `cdi_scenarios!` in
+`tests/suites/features/cdi/tests/injection.rs`. A new spec file belongs in
+`cdi_fixture_spec_files` in the role defaults, which the setup and test
+playbooks share.
+
+Directory absence, daemon configuration refresh, and restart tests require
+orchestration around these fixtures. Spec projection secrecy is a separate
+workload assertion. Hardware CUDA initialization remains outside this suite.
+
 Print the exact tmachine archive selection as a shell `PODMAN_CI_TESTS` array:
 
 ```shell

flake.nix

@@ -169,6 +169,10 @@
             type = "app";
             program = "${artifacts.testArchives}/bin/build-artifacts-test-archives";
           };
+          build-artifacts-cdi-tests = {
+            type = "app";
+            program = "${artifacts.cdiArchive}/bin/build-openshell-cdi-test-archive";
+          };
           build-artifacts-test-images = {
             type = "app";
             program = "${artifacts.testImages}/bin/build-artifacts-test-images";

tests/ansible/playbooks/features/cdi/tests.yaml

@@ -0,0 +1,105 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+---
+- name: Test CDI injection without GPU hardware
+  hosts: all
+  gather_facts: false
+  vars_files:
+    - ../../../roles/cdi_fixtures/defaults/main.yaml
+  vars:
+    cdi_test_root: /var/lib/openshell-feature-tests/cdi
+  tasks:
+    - name: Wait for SSH
+      ansible.builtin.wait_for_connection:
+
+    - name: Resolve installed gateway context
+      ansible.builtin.include_role:
+        name: openshell_test_gateway
+
+    - name: Require environment CDI specs after boot
+      ansible.builtin.stat:
+        path: "/var/run/cdi/{{ item }}"
+      loop: "{{ cdi_fixture_spec_files }}"
+      register: cdi_environment_specs
+
+    - name: Require CDI fixtures supplied by the Docker environment
+      ansible.builtin.assert:
+        that: [item.stat.isreg | default(false)]
+        fail_msg: "Docker environment did not recreate CDI spec {{ item.item }} after boot"
+      loop: "{{ cdi_environment_specs.results }}"
+
+    - name: Save gateway configuration before enabling caller device selection
+      become: true
+      ansible.builtin.slurp:
+        src: "{{ openshell_test_gateway.config_path }}"
+      register: cdi_gateway_config
+
+    - name: Run CDI tests with caller device selection enabled
+      block:
+        - name: Enable caller driver configuration for CDI tests
+          become: true
+          ansible.builtin.blockinfile:
+            path: "{{ openshell_test_gateway.config_path }}"
+            insertafter: '^\[openshell\.drivers\.docker\]$'
+            marker: "# {mark} OpenShell CDI test fixture"
+            block: allow_driver_config = true
+
+        - name: Restart gateway with caller device selection enabled
+          ansible.builtin.include_role:
+            name: openshell_test_gateway
+            tasks_from: restart.yaml
+
+        - name: Create CDI test directory
+          become: true
+          ansible.builtin.file:
+            path: "{{ cdi_test_root }}"
+            state: directory
+            owner: tmachine
+            group: tmachine
+            mode: "0700"
+
+        - name: Install CDI test archive
+          become: true
+          ansible.builtin.unarchive:
+            src: "{{ openshell_cdi_test_bundle }}"
+            dest: "{{ cdi_test_root }}"
+            owner: tmachine
+            group: tmachine
+
+        - name: Run CDI feature tests
+          ansible.builtin.command:
+            argv: [cargo-nextest, nextest, run, --archive-file, "{{ cdi_test_root }}/tests.tar.zst", --workspace-remap, "{{ cdi_test_root }}", --no-fail-fast, --no-capture]
+          environment:
+            OPENSHELL_GATEWAY: "{{ openshell_test_gateway.name }}"
+            OPENSHELL_BIN: "{{ openshell_test_cli.stdout }}"
+          register: cdi_test_result
+          changed_when: false
+          failed_when: false
+
+        - name: Show CDI test result
+          ansible.builtin.debug:
+            var: cdi_test_result
+
+        - name: Collect gateway diagnostics after test failure
+          ansible.builtin.include_role:
+            name: openshell_test_gateway
+            tasks_from: journal.yaml
+          when: cdi_test_result.rc != 0
+
+        - name: Require CDI test success
+          ansible.builtin.assert:
+            that: [cdi_test_result.rc == 0]
+            fail_msg: CDI feature tests failed
+
+      always:
+        - name: Restore gateway configuration
+          become: true
+          ansible.builtin.copy:
+            content: "{{ cdi_gateway_config.content | b64decode }}"
+            dest: "{{ openshell_test_gateway.config_path }}"
+
+        - name: Restart gateway with its original configuration
+          ansible.builtin.include_role:
+            name: openshell_test_gateway
+            tasks_from: restart.yaml

tests/artifacts.nix

@@ -98,6 +98,14 @@ let
     target = muslToolchain.target;
     output = "artifacts/test-archives/${muslToolchain.target}/provider-refresh-keycloak-tests.tar";
   };
+  cdiArchive = mkTestArchive {
+    name = "openshell-cdi";
+    workspacePath = "tests/suites/features";
+    manifestPath = "tests/suites/features/Cargo.toml";
+    package = "openshell-test-feature-cdi";
+    target = muslToolchain.target;
+    output = "artifacts/test-archives/${muslToolchain.target}/openshell-cdi-tests.tar";
+  };
   ociImageArchive = mkTestArchive {
     name = "oci-image";
     workspacePath = "tests/suites/features";
@@ -205,6 +213,7 @@ rec {
   inherit
     conformanceTestArchive
     providerRefreshKeycloakArchive
+    cdiArchive
     ociImageArchive
     podmanDriverArchive
     podmanE2eArchive
@@ -304,13 +313,15 @@ rec {
     runtimeInputs = [
       conformanceTestArchive
       providerRefreshKeycloakArchive
+      cdiArchive
       ociImageArchive
       podmanDriverArchive
       podmanE2eArchive
     ];
     text = ''
       build-openshell-conformance-test-archive
       build-provider-refresh-keycloak-test-archive
+      build-openshell-cdi-test-archive
       build-oci-image-test-archive
       build-podman-driver-test-archive
       build-podman-e2e-test-archive

tests/config.nix

@@ -170,6 +170,13 @@ let
           openshell_conformance_test_bundle = "../artifacts/test-archives/${muslTarget}/openshell-conformance-tests.tar";
         };
       }
+      {
+        name = "cdi";
+        playbooks = [ "ansible/playbooks/features/cdi/tests.yaml" ];
+        inputs = {
+          openshell_cdi_test_bundle = "../artifacts/test-archives/${muslTarget}/openshell-cdi-tests.tar";
+        };
+      }
       {
         name = "provider-refresh";
         playbooks = [ "ansible/playbooks/features/provider-refresh/keycloak.yaml" ];

tests/suites/features/Cargo.lock

@@ -3,4 +3,4 @@
 
 [workspace]
 resolver = "2"
-members = ["oci-image", "provider-refresh/keycloak"]
+members = ["cdi", "oci-image", "provider-refresh/keycloak"]