Skip to content

test: share fixture binaries and finish the nemoclaw contract tests - #12967

Merged
cv merged 2 commits into
v1from
test/shared-fixture-binaries
Oct 10, 2026
Merged

test: share fixture binaries and finish the nemoclaw contract tests#12967
cv merged 2 commits into
v1from
test/shared-fixture-binaries

Conversation

@cv

@cv cv commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

Part of #12876; closes #12879's remaining ELSEWHERE entries and advances #12878.

Failure

The nemoclaw provider's contract binary listed five types in ELSEWHERE, tested only in nemoclaw-e2e:

  • managed_gateway, service_capacity, and service_readiness tests needed nemoclaw-e2e's fixture provider and SSH simulator. Cargo sets CARGO_BIN_EXE_* only for a package's own tests, so the provider crate could not use them.
  • gateway_storage had no test without real Docker.
  • inference_capabilities was listed as tested only through SDK deployments, but nemoclaw-e2e already had an authored-HCL test for it.

Decision

  • Shared fixture executables. terraform-provider-nemoclaw-fixture and nemoclaw-e2e-ssh-fixture move into nemoclaw-test-fixtures as nemoclaw-fixture-provider and nemoclaw-fixture-ssh-simulator. fixture_executable(name) finds them in the parent of the test executable's deps directory. That is where Cargo writes binaries and where nextest extracts archived non-test binaries. Because the lifecycle filter already selects nemoclaw-test-fixtures' ssh_relay binary, archives include both new executables. I confirmed this with nextest archive. ssh::install_simulator and path_with replace nemoclaw-e2e's helpers. The simulator still relays loopback fixture engines on Windows (test: run the SSH simulator's tests on Windows #12963).

  • Moved tests. provider_protocol, service_capacity, and service_readiness move without changes to their assertions. remote_service and service_images in nemoclaw-e2e now get the simulator from nemoclaw-test-fixtures.

  • inference_capabilities. The authored-HCL test moves from nemoclaw-e2e. New cases cover an Anthropic catalog that rejects the request without a credential (unavailable, authentication required, no upstream text) and invalid inputs that fail validation without contacting the endpoint. The SDK-deployment test stays in nemoclaw-e2e.

  • gateway_storage. A new test uses a stateful fake Docker engine that stores volumes, networks, containers, and archive uploads. It checks:

    • validation of an invalid generation
    • a plan that changes nothing on the engine
    • one initialization: network, volume, initializer, key, and gateway.toml
    • the data_path output
    • an unchanged plan
    • refresh failures that keep state for configuration drift and a missing volume
    • a destroy that is refused while state is kept

    Managed gateway resources accept only a local engine socket, so this module is #[cfg(unix)] with a comment, like the existing managed-gateway test.

  • ELSEWHERE is now empty. The coverage test still enforces it.

  • Cost: nemoclaw-test-fixtures now depends on nemoclaw-sdk, nemoclaw-runtime (without default features), nemoclaw-tofu, and tf-provider, which the fixture provider needs. Building openshell-provider or fabric-provider tests alone now also compiles the SDK.

Validation

  • Before the move, every_type_has_a_contract_test failed (managed_gateway has a contract test; remove it from ELSEWHERE), and the moved tests failed because nemoclaw-fixture-provider was missing. Both pass after the move.
  • Ran the whole nemoclaw contract binary with --run-ignored all against a freshly built linux_arm64 bundle: 27 passed.
  • remote_service:: in nemoclaw-e2e (simulator through the bundle): 8 passed.
  • nemoclaw-test-fixtures tests, including ignored: 5 passed.
  • cargo fmt --all and cargo clippy --workspace --all-targets -- -D warnings are clean on Linux. Windows and macOS were not checked locally; CI covers them.

@cv
test: share fixture binaries and finish the nemoclaw contract tests
cfd1eeb 
Move nemoclaw-e2e's fixture provider and SSH simulator into
nemoclaw-test-fixtures, found beside the test executables, so the
provider_protocol, service_capacity, and service_readiness tests run in
the nemoclaw provider's contract binary. Add contract tests for
gateway_storage against a fake Docker engine and for
inference_capabilities against a fake model server, and empty ELSEWHERE.
@cv cv added area: e2e End-to-end tests, nightly failures, or validation infrastructure v1 NemoClaw v1 branch labels Oct 10, 2026
@cv
Merge remote-tracking branch 'origin/v1' into test/shared-fixture-bin…
26e643a 
…aries

# Conflicts:
#	crates/nemoclaw-e2e/tests/remote_service.rs
@github-actions

github-actions Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

v1 documentation preview: https://nvidia-preview-nemoclaw-v1-pr-12967.docs.buildwithfern.com/nemoclaw

@cv
cv merged commit b42d188 into v1 Oct 10, 2026
39 checks passed
@cv
cv deleted the test/shared-fixture-binaries branch October 10, 2026 17:01
This was referenced Oct 10, 2026
Closed
Open
Closed
Open
Merged
cv added a commit that referenced this pull request Oct 10, 2026
@cv
test: make the bundle the only test input and split out the fixture p…
29e4506 
…rovider (#12971)

Refs #12878 (the two parts still open).

## Failure

- Tests that run OpenTofu took up to three inputs: `NEMOCLAW_TEST_TOFU`
and `NEMOCLAW_TEST_PROVIDER` for the contract tests and two
`nemoclaw-e2e` tests, and `NEMOCLAW_TEST_BUNDLE` for the rest. The
bundle already ships pinned OpenTofu and all three providers. The
contract tests also found the `openshell` and `fabric` providers beside
`NEMOCLAW_TEST_PROVIDER`, so `cargo ci archive` had to pack
`target/debug` providers for the lifecycle workers.
- After #12967, `nemoclaw-test-fixtures` depended on `nemoclaw-sdk` and
`tf-provider` for the fixture provider. Building the `openshell` or
`fabric` provider tests therefore also compiled the SDK and its
Kubernetes, Fabric, and YAML dependencies.

## Decision

- `NEMOCLAW_TEST_BUNDLE` is now the only input.
`nemoclaw_test_fixtures::Bundle` finds the bundle's `libexec/tofu` and
each `providers/registry.opentofu.org/nvidia/NAME/VERSION/PLATFORM`
provider. `TofuWorkspace::new(&bundle, nemoclaw)` takes `openshell` and
`fabric` from the bundle.
- Each provider's contract tests still test the provider built from the
checkout, with no separate input. `package_executable!` reads nextest's
`NEXTEST_BIN_EXE_<name>`, which also works after archive extraction, and
falls back to Cargo's `CARGO_BIN_EXE_<name>` under `cargo test`. Any
other provider those tests need comes from the bundle. The two
`nemoclaw-e2e` tests (`sandbox_readiness`, `managed`) run the bundle's
`nemoclaw` provider.
- `cargo ci lifecycle` sets only `NEMOCLAW_TEST_BUNDLE`.
`lifecycle-inputs.tar` keeps only the fake `ssh` relay, which the
Windows lifecycle step puts on PATH. Nextest already archives each
selected package's executables. `ssh_relay()` no longer looks beside the
providers.
- The fixture provider moves to a new `nemoclaw-fixture-provider` crate.
An ignored test that the lifecycle profile selects makes nextest archive
it, as `ssh_relay` does for the relay.
- Updated `docs/contributing/integration-tests.md` and `testing.md`.

Build cost of `cargo build --locked -p openshell-provider --tests` from
a clean target directory, measured sequentially on the same aarch64 host
while other jobs ran (load 2–15):

| | Crates compiled | Wall time |
|---|---|---|
| `origin/v1` (b42d188) | 374 | 67 s |
| This branch | 324 | 47 s |

The 50 crates no longer compiled include `nemoclaw-sdk`,
`nemoclaw-fabric`, `nemo-fabric-core`, `kube`, `k8s-openapi`, `jiff`,
`toml`, `pest`, and `serde-saphyr`. Wall time is noisy on a shared host;
the crate count is deterministic.

## Validation

- `ci_lifecycle`:
`lifecycle_partition_reaches_nextest_without_changing_the_selected_suite`
now requires that only an absolute `NEMOCLAW_TEST_BUNDLE` reaches
nextest, and the archive test requires that no provider is packed. I
watched both fail before the change; all 7 tests pass now.
- New `bundle` unit tests in `nemoclaw-test-fixtures` pass.
- I built a fresh `linux_arm64` bundle, unset `NEMOCLAW_TEST_TOFU` and
`NEMOCLAW_TEST_PROVIDER`, and set only `NEMOCLAW_TEST_BUNDLE`. Under
nextest with `--run-ignored all`, these pass:
  - `openshell-provider` and `fabric-provider` `contract` (50 tests)
  - `nemoclaw-provider` `contract` (27 tests)
  - `nemoclaw-e2e` `sandbox_readiness::`
  - the new `nemoclaw-fixture-provider` archive test
  - `nemoclaw-test-fixtures`
- Archive handoff: `cargo nextest archive` with
`binary_id(=openshell-provider::contract)` included
`terraform-provider-openshell`. With
`target/debug/terraform-provider-openshell` moved away, the archived
`capabilities::` tests passed from `--extract-to`, so they used the
extracted provider.
- `cargo test -p openshell-provider --test contract capabilities:: --
--ignored` passes through the `CARGO_BIN_EXE` fallback. Without the
variable, tests fail with `NEMOCLAW_TEST_BUNDLE names a bundle; build
one with cargo ci bundle`.
- `cargo fmt --all` and `cargo clippy --workspace --all-targets -- -D
warnings` are clean on Linux. I did not run Windows or macOS locally; CI
covers them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure v1 NemoClaw v1 branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cv