Skip to content

Commit 9b52b43

Browse files
SDAChesselezar
SDAChess
and
elezar
authored
test(tmachine): add portable VM-based container runtime testing (#3371)
* test(tmachine): add Docker VM scenario Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(tmachine): add portable container scenarios Signed-off-by: Simon Scatton <sscatton@nvidia.com> * chore(tmachine): isolate downloaded Ansible roles Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(tmachine): increase VM resources Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(tmachine): improve artifact builds and diagnostics Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(nix): pin tmachine runtime on macOS Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(tests): enable tty and fix supervisor image path * fix(nix): isolate testing tools from default shell Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(tmachine): initialize test runner working directory Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(tmachine): include Ansible sources in layer cache keys Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(tmachine): build and load separate runtime images Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(tmachine): use local runtime images for Docker Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com>
1 parent 4a3f267 commit 9b52b43

41 files changed

Lines changed: 2363 additions & 6 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,9 @@ temp/
186186
*.tmp
187187
e2e/gpu/images/.build/
188188

189+
# Ansible Galaxy roles
190+
/tests/ansible/.roles/
191+
189192
# Secrets/credentials (should never be committed)
190193
*.pem
191194
*.key

‎architecture/build.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,8 @@ The Nix test guest harness under `nix/test-guest` boots native-architecture clou
227227
through QEMU for package, release, and E2E validation. A prepared cache entry is
228228
captured after the exact ordered Ansible configuration list and before
229229
test-specific packages, copied binaries, forwarded ports, or commands.
230+
On macOS, the test guest and tmachine paths use the same pinned QEMU and OVMF
231+
package set so the hypervisor and firmware remain compatible.
230232

231233
Prepared disks are flattened, sanitized QCOW2 images. The local cache keeps them
232234
read-only and each test receives a fresh writable overlay and cloud-init
@@ -242,6 +244,23 @@ CLI conformance runs after target provisioning and operates only through the
242244
configured OpenShell CLI. The smoke scenario verifies the black-box sandbox
243245
lifecycle by creating, inspecting, executing in, and deleting a sandbox.
244246

247+
The `tests/tmachine` setup and installation caches include a digest of the
248+
entire directory containing `ANSIBLE_CONFIG`, including local roles, task
249+
includes, templates, inventory, and requirements. The digest uses sorted
250+
relative paths, file contents, and executable permissions; source symlinks
251+
are unsupported. Both keys also retain the ordered playbook paths and contents,
252+
their base disk contents, and whether Galaxy is enabled; installation keys
253+
include named binary inputs. The top-level `.roles` directory is excluded:
254+
Galaxy release pins in `requirements.yaml` are treated as immutable, including
255+
any transitive dependency pins. Cache misses with Galaxy enabled reinstall
256+
the required roles and their dependencies before running playbooks.
257+
258+
The `tests/artifacts.nix` helpers build the CLI, conformance CLI, and sandbox
259+
with musl, and the gateway and supervisor with GNU. Image assembly stages
260+
the gateway, sandbox, and supervisor as separate binaries for their respective
261+
Dockerfiles. The Ubuntu Docker and Fedora Podman scenarios import both local
262+
runtime images and configure the gateway to use them.
263+
245264
## Python Wheel Packaging
246265

247266
The generated protobuf/gRPC stubs under `python/openshell/_proto/` are gitignored

‎flake.nix‎

Lines changed: 53 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
overlays = [ (import rust-overlay) ];
4646
};
4747
testGuestPkgs = import nixpkgs-test-guest { inherit system; };
48+
tmachineRuntimePkgs = if pkgs.stdenv.hostPlatform.isDarwin then testGuestPkgs else pkgs;
4849
commonDevShellPackages = with pkgs; [
4950
actionlint
5051
cargo-auditable
@@ -67,6 +68,10 @@
6768
zizmor
6869
zstd
6970
];
71+
commonDevShell = {
72+
packages = [ rustToolchain ] ++ commonDevShellPackages;
73+
env = pkgs.lib.foldl' (env: toolchain: env // toolchain.env) { } (builtins.attrValues toolchains);
74+
};
7075
treefmtEval = treefmt-nix.lib.evalModule pkgs {
7176
projectRootFile = "flake.nix";
7277
programs.nixfmt.enable = true;
@@ -114,17 +119,59 @@
114119
qemuPkgs = testGuestPkgs;
115120
firmwarePkgs = testGuestPkgs;
116121
};
122+
testMachines = import ./tests/config.nix {
123+
inherit pkgs toolchains;
124+
qemuPkgs = tmachineRuntimePkgs;
125+
firmwarePkgs = tmachineRuntimePkgs;
126+
};
127+
artifacts = pkgs.callPackage ./tests/artifacts.nix { inherit rustToolchain toolchains; };
117128
in
118129
{
119-
apps.test-guest = testGuest.app;
120-
apps.test-guest-cache = testGuest.cacheApp;
130+
apps = {
131+
build-artifacts = {
132+
type = "app";
133+
program = "${artifacts.all}/bin/build-artifacts";
134+
};
135+
build-artifacts-binaries = {
136+
type = "app";
137+
program = "${artifacts.binaries}/bin/build-artifacts-binaries";
138+
};
139+
build-artifacts-helm = {
140+
type = "app";
141+
program = "${artifacts.helm}/bin/build-artifacts-helm";
142+
};
143+
build-artifacts-images = {
144+
type = "app";
145+
program = "${artifacts.images}/bin/build-artifacts-images";
146+
};
147+
test-guest = testGuest.app;
148+
test-guest-cache = testGuest.cacheApp;
149+
};
150+
151+
packages = {
152+
vm-runtime = vmRuntime;
153+
tmachine = testMachines.package;
154+
tmachine-config = testMachines.config;
155+
tmachine-unwrapped = testMachines.unwrapped;
156+
};
121157

122-
packages.vm-runtime = vmRuntime;
158+
devShells = {
159+
default = pkgs.mkShellNoCC commonDevShell;
123160

124-
devShells.default = pkgs.mkShellNoCC {
125-
packages = [ rustToolchain ] ++ commonDevShellPackages;
161+
testing = pkgs.mkShellNoCC (
162+
commonDevShell
163+
// {
164+
packages = commonDevShell.packages ++ [
165+
pkgs.ansible
166+
pkgs.sshpass
167+
testMachines.package
168+
];
126169

127-
env = pkgs.lib.foldl' (env: toolchain: env // toolchain.env) { } (builtins.attrValues toolchains);
170+
shellHook = ''
171+
export ANSIBLE_CONFIG="$(git rev-parse --show-toplevel)/tests/ansible/ansible.cfg"
172+
'';
173+
}
174+
);
128175
};
129176

130177
formatter = treefmtEval.config.build.wrapper;

‎tests/ansible/ansible.cfg‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[defaults]
5+
inventory = inventory.ini
6+
host_key_checking = False
7+
roles_path = .roles:roles
8+
callbacks_enabled = ansible.posix.profile_tasks
9+
10+
[callback_profile_tasks]
11+
summary_only = true
12+
task_output_limit = 20
13+
sort_order = descending
14+
15+
[ssh_connection]
16+
ssh_common_args = -o UserKnownHostsFile=/dev/null

‎tests/ansible/inventory.ini‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[machines]
5+
tmachine ansible_host=127.0.0.1 ansible_port=2222 ansible_user=tmachine ansible_password=tmachine ansible_python_interpreter=/usr/bin/python3

‎tests/ansible/playbooks/docker.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Docker
6+
hosts: all
7+
gather_facts: false
8+
vars:
9+
docker_users:
10+
- tmachine
11+
pre_tasks:
12+
- name: Wait for SSH
13+
ansible.builtin.wait_for_connection:
14+
15+
- name: Gather facts
16+
ansible.builtin.setup:
17+
roles:
18+
- role: geerlingguy.docker
19+
become: true

‎tests/ansible/playbooks/gateway.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Configure OpenShell gateway
6+
hosts: all
7+
gather_facts: false
8+
roles:
9+
- openshell_gateway

‎tests/ansible/playbooks/openshell.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Install OpenShell
6+
hosts: all
7+
gather_facts: false
8+
tasks:
9+
- name: Wait for SSH
10+
ansible.builtin.wait_for_connection:
11+
12+
- name: Install OpenShell CLI
13+
become: true
14+
ansible.builtin.copy:
15+
src: "{{ openshell_cli_binary }}"
16+
dest: /usr/local/bin/openshell
17+
mode: "0755"
18+
19+
- name: Install OpenShell gateway
20+
become: true
21+
ansible.builtin.copy:
22+
src: "{{ openshell_gateway_binary }}"
23+
dest: /usr/local/bin/openshell-gateway
24+
mode: "0755"

‎tests/ansible/playbooks/podman-rootful.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Configure rootful Podman
6+
hosts: all
7+
gather_facts: false
8+
tasks:
9+
- name: Wait for SSH
10+
ansible.builtin.wait_for_connection:
11+
12+
- name: Enable rootful Podman socket
13+
become: true
14+
ansible.builtin.systemd_service:
15+
name: podman.socket
16+
enabled: true
17+
state: started

‎tests/ansible/playbooks/podman-rootless.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
---
5+
- name: Configure rootless Podman
6+
hosts: all
7+
gather_facts: false
8+
tasks:
9+
- name: Wait for SSH
10+
ansible.builtin.wait_for_connection:
11+
12+
- name: Enable lingering for tmachine
13+
become: true
14+
ansible.builtin.command:
15+
argv:
16+
- loginctl
17+
- enable-linger
18+
- tmachine
19+
changed_when: false
20+
21+
- name: Resolve tmachine UID
22+
ansible.builtin.command:
23+
argv:
24+
- id
25+
- -u
26+
- tmachine
27+
changed_when: false
28+
register: tmachine_uid
29+
30+
- name: Start tmachine user manager
31+
become: true
32+
ansible.builtin.systemd_service:
33+
name: "user@{{ tmachine_uid.stdout }}.service"
34+
state: started
35+
36+
- name: Enable rootless Podman socket
37+
become: true
38+
become_user: tmachine
39+
ansible.builtin.systemd_service:
40+
name: podman.socket
41+
scope: user
42+
enabled: true
43+
state: started
44+
environment:
45+
HOME: /home/tmachine
46+
XDG_RUNTIME_DIR: "/run/user/{{ tmachine_uid.stdout }}"
47+
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ tmachine_uid.stdout }}/bus"

0 commit comments

Comments
 (0)