Skip to content

test: build the Kubernetes lifecycle workspace from the bundle - #12981

Merged
cv merged 1 commit into
v1from
fix/kubernetes-lifecycle-bundle
Oct 11, 2026
Merged

test: build the Kubernetes lifecycle workspace from the bundle#12981
cv merged 1 commit into
v1from
fix/kubernetes-lifecycle-bundle

Conversation

@cv

@cv cv commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Failure

v1 doesn't compile since #12972. Its new nemoclaw-provider/tests/contract/kubernetes_lifecycle.rs calls TofuWorkspace::new(tofu, provider), with OpenTofu and the provider read from NEMOCLAW_TEST_TOFU and NEMOCLAW_TEST_PROVIDER. #12971 merged first and changed that function to TofuWorkspace::new(&Bundle, nemoclaw). Since #12971, cargo ci lifecycle also no longer sets either variable. Each PR passed CI against a v1 without the other.

Every PR's CI now fails at Lint Rust:

error[E0308]: mismatched types
   --> crates/nemoclaw-provider/tests/contract/kubernetes_lifecycle.rs:111:44
    |
111 |         let workspace = TofuWorkspace::new(tofu, provider);
    |                                            ^^^^ expected `&Bundle`, found `PathBuf`

Decision

Build the workspace with crate::workspace(), as the crate's other contract tests do since #12971: OpenTofu and the openshell and fabric providers from NEMOCLAW_TEST_BUNDLE, and the nemoclaw provider built from the checkout. That drops the two environment variables and the now-unused PathBuf import.

Validation

  • cargo clippy -p nemoclaw-provider --tests -- -D warnings is clean.
  • cargo ci passed: 1,331 workspace tests and 131 lifecycle tests. Both of the file's tests ran from the bundle and passed:
    • kubernetes_resources_apply_observe_the_release_and_tear_down_keeping_storage
    • kubernetes_resources_refuse_replaced_or_missing_objects_without_recreating_them

@cv cv mentioned this pull request Oct 11, 2026
Merged
@github-actions

github-actions Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

v1 documentation preview: https://nvidia-preview-nemoclaw-v1-pr-12981.docs.buildwithfern.com/nemoclaw

cv added a commit that referenced this pull request Oct 11, 2026
@cv
test(provider): build the Kubernetes lifecycle workspace from the bundle
a574d7e 
v1 at a49fc77 does not compile the provider contract tests:
kubernetes_lifecycle.rs still passes paths to TofuWorkspace::new, which
now takes the bundle. This is the same change as #12981, so CI can run
the lifecycle tests on this branch; merging v1 after #12981 lands
leaves one copy.
cv added a commit that referenced this pull request Oct 11, 2026
@cv
test(e2e): read and remove remote service networks less often (#12982)
8d34abd 
Part of #12882.

## Failure

The pinned `kreuzwerker/docker` provider waits 2 s before every
`docker_network` read and after every removal. Every runtime plan,
apply, export, or destroy reads the network once it exists, and a
destroy also waits for the removal. Only the `remote_service` lifecycle
tests deploy a service with a Docker network: the SSH simulator is the
only fake engine that serves `/networks`.

I counted the network reads per test from a request log in the SSH
simulator (local `linux_arm64` bundle; the logging is not part of this
change). The 12 tests made 75 reads, about 150 s of their 404 s of test
time:

| Test | Reads |
|---|---|
|
`managed_bearer_storage_or_key_damage_stops_plan_and_apply_without_changing_bindings`
| 13 |
|
`remote_model_reapply_and_drift_keep_bindings_and_stop_on_observation_failure`
| 11 |
|
`remote_ollama_reapply_and_drift_keep_bindings_and_stop_on_observation_failure`
| 11 |
|
`managed_bearer_credentials_survive_recovery_reapply_and_replacement_outside_state`
| 10 |
|
`managed_pi_applies_without_generation_and_refused_sandbox_changes_keep_intent`
| 9 |
|
`managed_bearer_replacement_cleanup_recovers_and_teardown_accepts_both_identities`
| 7 |
|
`remote_model_apply_recovers_failed_creation_and_lost_process_or_cache_keeping_data`
| 7 |
|
`remote_ollama_apply_recovers_failed_creation_and_lost_process_or_cache_keeping_data`
| 7 |
| partial-destroy and pulled-image tests (4) | 0 |

Several tests created and destroyed a runtime only to repeat checks that
another scenario already ran against the same kind of runtime.

## Decision

Run each check against one runtime per scenario, and only once. The
Docker provider is unchanged.

| Before | After |
|---|---|
|
`remote_model_apply_recovers_failed_creation_and_lost_process_or_cache_keeping_data`
| Removed. vLLM failed-creation, startup, lost-process, and cache
recovery run in
`managed_bearer_credentials_survive_recovery_reapply_and_replacement_outside_state`,
against the same vLLM runtime with a bearer. Present-image compatibility
runs in the Ollama and Pi tests, and the pull-policy plan runs in the
Ollama test. |
| `remote_ollama_apply_recovers_…` and
`remote_ollama_reapply_and_drift_…` |
`remote_ollama_recovers_reapplies_and_stops_on_observation_failure_keeping_data`:
recovery, then export and reapply, the transport failure, and teardown
against the recovered runtime. The readiness gate, compute replacement
plan, and pull-policy reapply do not depend on the runtime kind and run
against vLLM in `remote_model_reapply_and_drift_…`.
`managed_installers_accept_current_runtime_readiness_without_collecting_model_files`
in `nemoclaw-provider` covers each runtime's readiness observation. |
| `managed_bearer_storage_or_key_damage_…` and
`managed_bearer_replacement_cleanup_…` |
`managed_bearer_cleanup_recovers_damage_keeps_bindings_and_teardown_accepts_both_identities`:
the daemon retargeting, credential-volume damage, and insecure-key
checks run against the state that replacement cleanup settles, before
the teardown with both identities. |
| Missing, foreign, and substituted credential volumes through CLI plan
and apply | New
`service_storage::missing_foreign_or_substituted_storage_stops_plan_and_apply_without_changing_state`
contract test, for both storage kinds, through pinned OpenTofu against a
fake engine. The lifecycle test keeps one substituted volume to show the
CLI stops before changing bindings. |
| Pi: failed-creation recovery, then an export after the last refusal |
Pi applies once; the bearer and Ollama tests own recovery. The export
after the last refusal is dropped: the intent is byte-identical to the
one the earlier export already reproduced, and the destroy that follows
reads the same state. |

The partial-destroy and pulled-image tests are unchanged.

## Validation

- Local `linux_arm64` run of `remote_service`, same request log:

  | | Tests | Network reads | Summed test time | Wall (4 threads) |
  |---|---|---|---|---|
  | Before | 12 | 75 | 404 s | 110 s |
  | After | 9 | 48 | 280 s | 97 s |

After the change, by test: vLLM reapply 11, Ollama 9, bearer credentials
10, bearer cleanup and damage 12, Pi 6.
- The new contract test asserts each refusal reason (`bound persistent
storage is absent; recreation forbidden`, `observed ownership,
generation, or durable identity changed`).
- `remote_service` and `service_storage` lifecycle tests pass locally;
`cargo fmt --all --check` and `cargo clippy --workspace --all-targets --
-D warnings` are clean.
- `v1` at a49fc77 does not compile the provider contract tests
(`kubernetes_lifecycle.rs`). This branch carries the same fix as #12981
in its own commit so CI can run, and merging `v1` after #12981 lands
leaves one copy.
- CI lifecycle JUnit reports from run
[38100630144](https://github.com/NVIDIA/NemoClaw/actions/runs/38100630144),
compared with the `v1` run for #12974
([38098264065](https://github.com/NVIDIA/NemoClaw/actions/runs/38098264065)),
the latest `v1` run whose lifecycle tests ran. Cells are partition wall
seconds / summed test seconds / `remote_service` summed seconds:

  | Partition | Before | After |
  |---|---|---|
  | linux_arm64 / 1 | 198.8 / 788 / 175 | 200.2 / 795 / 171 |
  | linux_arm64 / 2 | 178.0 / 711 / 252 | 149.3 / 587 / 120 |
  | linux_amd64 / 1 | 221.0 / 878 / 209 | 208.0 / 825 / 186 |
  | linux_amd64 / 2 | 174.0 / 696 / 247 | 165.6 / 654 / 142 |

`remote_service` went from 427 s to 291 s summed on linux_arm64, and
from 456 s to 328 s on linux_amd64. Across the 9 `Rust desired-state`
runs before #12974, it ranged 446–454 s on linux_arm64. The network read
counts above come from the local request log; CI runs the same commands.
The linux_arm64 lifecycle step is still bounded by partition 1, which
kept its share of the `remote_service` tests and did not get shorter;
partition 2 dropped 29 s.
@cv
cv merged commit 18748d2 into v1 Oct 11, 2026
75 of 77 checks passed
@cv
cv deleted the fix/kubernetes-lifecycle-bundle branch October 11, 2026 09:57
cv added a commit that referenced this pull request Oct 11, 2026
@cv
ci: balance lifecycle partitions with nextest slices (#12991)
fea22ba 
## Failure

Lifecycle tests are split into partitions with nextest's `hash:` mode.
With 129 tests, the hash puts 74 of them in partition 1 and 55 in
partition 2 on Linux and macOS. On Windows the three partitions get 35,
41, and 39 tests. Each partition keeps its four test slots busy, so its
wall time is about its summed test time divided by four, and the slowest
partition sets the step's wall time.

Execute step times on recent v1 runs, in seconds:

| Run | linux_arm64 | linux_amd64 | darwin_arm64 | windows_amd64 |
|---|---|---|---|---|
| [#12981](https://github.com/NVIDIA/NemoClaw/actions/runs/38130235163)
| 208 / 158 | 203 / 157 | 187 / 145 | 105 / 176 / 152 |
| [#12982](https://github.com/NVIDIA/NemoClaw/actions/runs/38129638172)
| 208 / 158 | 233 / 173 | 215 / 139 | 84 / 145 / 156 |
| [#12974](https://github.com/NVIDIA/NemoClaw/actions/runs/38098264065)
| 207 / 186 | 231 / 184 | 183 / 175 | 113 / 169 / 227 |

## Decision

Use `--partition slice:m/n`. Nextest sorts the selected tests by binary
and name and deals them round-robin. The slow scenarios are
`nemoclaw-e2e` `deployment::` and `remote_service::` tests, and they
sort next to each other, so dealing them alternately splits them evenly.
Each partition now gets 64 or 65 tests on Linux.

I replayed the per-test times from two v1 runs' JUnit reports through a
four-slot scheduler that follows nextest's order. For the hash
partitions, it reproduces the measured partition times to within about 1
s. For linux_arm64, I checked the slice membership against `cargo
nextest list --partition slice:m/2` on that run's CI archive. Predicted
nextest wall time, in seconds:

| Platform | Hash, slowest partition | Slice, per partition | Ideal (sum
/ slots) |
|---|---|---|---|
| linux_arm64 | 201 | 177 / 174 | 174 |
| linux_amd64 | 193–222 | 170 / 170 – 193 / 193 | 169–191 |
| darwin_arm64 | 178–205 | 160 / 150 – 172 / 165 | 152–163 |
| windows_amd64 | 142–164 | 128 / 141 / 147 – 127 / 135 / 120 | 105–120
|

Alternatives considered:

- Test priorities that start the `remote_service::` and `deployment::`
scenarios first would only help Windows, by about 15 s on top of slices.
They would also tie the nextest config to module names, so I left them
out.
- A third Linux or macOS partition would bring each one to about 120–135
s. It would also add a runner and about 20 s of job setup and extraction
per partition. On Linux, lifecycle is not on the critical path: Live /
Docker and Live / Kind start at the same time and run for 490–750 s.
- Two Windows partitions would take about 170–187 s, which is slower
than three hash partitions are today.

Trade-off: with slice partitions, membership depends on the selected
test list. Adding or removing a lifecycle test moves later tests to
other partitions. For a given revision and platform, partitions stay
fixed.

Part of #12882.

## Validation

- Locally: `cargo fmt --all --check` and `cargo clippy --workspace
--all-targets -- -D warnings`. The change is to CI configuration and
docs, with no Rust behavior to test;
`crates/nemoclaw-build/tests/ci_lifecycle.rs` already covers passing
`--partition` through.
- `cargo nextest list --profile lifecycle --run-ignored only --partition
hash:m/2` on run 38130235163's linux_arm64 archive lists 74 / 55 tests,
matching CI. `slice:m/2` lists 65 / 64.
- [CI on this
PR](https://github.com/NVIDIA/NemoClaw/actions/runs/38133026550): every
lifecycle partition passed. Execute step times in seconds, with nextest
wall time and summed test time from the JUnit reports:

| Platform | Before, slowest step (3 v1 runs) | After, steps | After,
nextest wall | After, summed test time |
|---|---|---|---|---|
| linux_arm64 | 207–208 | 184 / 184 | 175 / 175 | 695 / 694 |
| linux_amd64 | 203–233 | 202 / 207 | 190 / 196 | 757 / 777 |
| darwin_arm64 | 183–215 | 159 / 145 | 151 / 137 | 578 / 542 |
| windows_amd64 | 156–227 | 143 / 131 / 152 | 132 / 118 / 142 | 441 /
425 / 499 |

This run's linux_amd64 tests summed to 1,534 s. That is close to the
1,528 s of the #12982 run, whose slowest hash partition took 233 s; the
slice partitions finished within 6 s of each other.
@cv cv added the v1 NemoClaw v1 branch label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

v1 NemoClaw v1 branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cv