Skip to content

Test in three layers with wall-clock budgets #12876

Closed
8 of 8 issues completed
Closed
Test in three layers with wall-clock budgets#12876
8 of 8 issues completed
Labels
area: ciCI workflows, checks, release automation, or GitHub Actionsarea: e2eEnd-to-end tests, nightly failures, or validation infrastructurev1NemoClaw v1 branch

Description

@cv
cv
opened on Oct 9, 2026

Problem

Unit, integration, and end-to-end tests check the same behavior at several layers, yet regressions still reach v1:

Cost is wall-clock time, not code. On linux_arm64 (CI / Native), deterministic tests take 62 s, and the lifecycle step takes 684 s for 109 tests that sum to 2,673 s. Three files account for 82% of that sum: deployment (1,247 s), remote_service (608 s), and fabric_deployment (335 s).

Layers

Layer Answers Lives in Budget on Linux CI runners
Unit Is the rule right? Each crate's src/, in-process fakes only Workspace step ≤ 90 s; each test ≤ 2 s
Contract Does each provider type work through pinned OpenTofu? One contract test binary per provider crate, with a fixture per type Each provider ≤ 90 s; each fixture ≤ 15 s
System Does SDK and CLI orchestration work? nemoclaw-e2e Lifecycle step ≤ 8 min; each scenario ≤ 120 s

Live Docker and Kind suites stay separate and report their time against 10 and 8 minutes.

A layer asserts only behavior its crate owns: SDK tests assert the generated configuration, provider tests assert their types, and no test decodes another crate's internal JSON.

Order

  1. Enforce wall-clock budgets per test layer #12877 records today's times and enforces the budgets, so every later change is measured.
  2. Share one test harness driven by the bundle #12878 gives every layer one harness.
  3. Contract tests for the nemoclaw provider #12879 and Contract tests for the openshell and fabric providers #12880 add contract tests per provider.
  4. Move tests to the layer that owns their behavior #12881 moves the remaining tests to the layer that owns them.
  5. Bring the system layer under its budget #12882 brings the system layer under budget.
  6. Run live suites on provider changes #12883 runs the live suites before merge.

Done when

All sub-issues are closed, CI enforces each budget, and the testing guide describes the layers and budgets.

Activity

  1. cv
    added
    area: ciCI workflows, checks, release automation, or GitHub Actions
    area: e2eEnd-to-end tests, nightly failures, or validation infrastructure
    v1NemoClaw v1 branch
    on Oct 9, 2026
  2. cv
    added sub-issues on Oct 9, 2026

  3. cv commented on Oct 9, 2026

    @cv
    cv
    on Oct 9, 2026
    CollaboratorAuthor

    Where each behavior is tested

    The map below uses linux_amd64 timings from #12885's CI run, after that PR's reductions. Unit counts come from test names and are approximate.

    Layer Tests Summed time
    Unit 1,300 189 s (24 s wall)
    Contract (pinned OpenTofu, one type) about 44 about 215 s
    System (SDK and CLI over a bundle) about 67 about 1,690 s
    Live (Docker, Kind) about 15 6–12 min per suite, on pushes to v1 only
    Behavior Unit Contract System
    Config parsing and schema about 400 tests none none
    Compile and graph shape about 100 tests vllm_runtime, kubernetes_hcl Implicit in every lifecycle
    Generic lifecycle: export reproduces the document, an unchanged reapply is a no-op, destroy cleans up SDK export and state tests Reference graph (63 s) About 35 tests, about 1,000 s: 15 configuration lifecycles in deployment, 10 Fabric harnesses, 4 remote lifecycles, multiple_providers, export tests
    Drift detection Provider observation tests opentofu_openshell, provider_protocol Inside each configuration lifecycle
    Failure and recovery Adapter and backend tests opentofu_openshell lost replies and untaint, service_storage, provider_protocol About 18 deployment scenarios, about 470 s
    Teardown and retention Provider retention tests opentofu_openshell, gateway_readiness Ends about 60 tests; about 8 test it directly
    Credentials and redaction SDK process redaction rules service_readiness cli_redaction, bearer lifecycle (131 s)
    Readiness Provider readiness and status service_readiness, gateway_readiness, sandbox_readiness apply_health_failure, readiness_and_observation_failures, remote lifecycles
    Images runtime_image unit tests runtime_image, provider_protocol unverified_*_images, remote pull checks
    Gateway compatibility nemoclaw-openshell observation reasons Reference graph incompatible_gateway…, gateway_change_between_plan_and_apply
    Fabric harnesses fabric_config, execution_settings none 10 harness tests (about 120 s)
    Kubernetes and Helm About 150 tests kubernetes_hcl (plan only) Helm recovery inside nemoclaw-sdk (5 tests, about 61 s); Kind live
    CLI output contract About 60 cli tests none cli_terminal_outputs… (37 s), startup failure messages

    Findings

    1. The generic lifecycle is checked about 35 times. Each repetition takes about 30 s on CI, about 60% of system time. What differs between variants is mostly what they compile to, which unit tests already own, plus one drift check. Planned change: keep one full lifecycle per gateway kind (external, Docker, Podman, Kubernetes). Each configuration variant instead gets a unit test that its compiled graph round-trips through export, using an SDK helper that exports from synthesized state without OpenTofu.
    2. Destroy is asserted about 60 times because it is the cleanup step. About 8 tests check teardown itself.
    3. The contract layer is lopsided. openshell has 23 contract tests, the nemoclaw provider about 20 scattered across 10 modules, Fabric none, and Kubernetes a plan-only check. Failure and recovery for nemoclaw and fabric types is therefore checked only in 30-second system scenarios. Contract tests for the nemoclaw provider #12879 and Contract tests for the openshell and fabric providers #12880 cover this.
    4. Failure and recovery overlaps across layers, but the system scenarios test SDK coordination: interrupted creates, checkpoints, refused recreation. They belong in the system layer; making each one cheaper is the remaining lever.
    5. Gaps where only expensive layers test a behavior:
      • Kubernetes apply and refresh have no contract test.
      • Helm recovery runs only inside the SDK crate and on Kind.
      • Fabric harness reconciliation is checked only at the system layer.
      • The CLI's lifecycle output is checked only in system tests.
  4. cv
    added 4 commits that reference this issue on Oct 9, 2026

    test: keep one system lifecycle per drift kind (#12906)

    2577138

    test: split tests into layers with wall-clock budgets (#12885)

    e2acbd9

    ci: fail a test step that exceeds its wall-clock budget (#12938)

    8ca6da3

    ci: run the live suites on pull requests that can affect them (#12937)

    7d5dd6e

  5. 1 remaining item

  6. cv
    added 7 commits that reference this issue on Oct 10, 2026

    feat(docker): reach SSH container engines from Windows (#12945)

    5f56a4d

    test(provider): move single-type nemoclaw tests into the provider's c…

    c91a37c

    test: reach fake engines from OpenTofu-launched providers on every pl…

    99206c8

    test: run the bundle-driven system tests on Windows (#12960)

    a19c6fa

    test: run the SSH simulator's tests on Windows (#12963)

    e2a70c0

    test: split the slow remote-service lifecycle scenarios (#12965)

    6a95532

    test: share fixture binaries and finish the nemoclaw contract tests (#…

    b42d188

  7. cv commented on Oct 10, 2026

    @cv
    cv
    on Oct 10, 2026
    CollaboratorAuthor

    Status on v1:

    Sub-issue Status
    #12877 Budgets Enforced on the Linux runners by #12938 and #12964. Open PR #12969 lowers the lifecycle budget and closes it.
    #12878 Shared harness Open. See its status comment.
    #12879 nemoclaw contract tests Open. Kubernetes apply and refresh need a fake Kubernetes API.
    #12880 openshell and fabric contract tests Open. #12908 and #12913 added the contract binaries and coverage tests.
    #12881 Move tests to their layer Open. See its status comment.
    #12882 System budget Open. See its status comment.
    #12883 Live suites on pull requests Closed. Done in #12937.

    Besides the sub-issues, two Done-when items remain:

    • .config/test-budgets.yaml has no contract-layer budget. Contract tests run inside the lifecycle partitions.
    • docs/contributing/testing.md lists the step budgets but doesn't describe the three layers.
  8. cv
    added 5 commits that reference this issue on Oct 10, 2026

    ci: lower the lifecycle budget to its measured times (#12969)

    11bba1d

    test: apply, refresh, and tear down the Kubernetes resources in the n…

    a49fc77

    ci: budget each provider's contract tests (#12990)

    67cac31

    ci: stop a contract test near its time limit (#12994)

    fd1ffb8

    docs: describe the test layers and their budgets (#12997)

    9970194

  9. cv commented on Oct 12, 2026

    @cv
    cv
    on Oct 12, 2026
    CollaboratorAuthor

    Done on v1.

    Since this issue was opened, on linux_amd64:

    Measure Before Now
    Deterministic step, wall time 78 s about 25 s
    Lifecycle one 744 s run two partitions of about 180–195 s
    Slowest lifecycle test 241 s about 71 s

    On Windows, the lifecycle step went from 52 to about 128 tests.

    #13005 tracks the remaining rewrite of model_live, a manual GPU test.

    v1 is not the default branch, so closing keywords did not apply.

  10. cv
    closed this as completedon Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciCI workflows, checks, release automation, or GitHub Actionsarea: e2eEnd-to-end tests, nightly failures, or validation infrastructurev1NemoClaw v1 branch

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions