Skip to content

ci: report each platform's whole lifecycle suite and budget the live suites - #12964

Merged
cv merged 4 commits into
v1from
ci/combined-timing-reports
Oct 10, 2026
Merged

cv merged 4 commits into
v1from
ci/combined-timing-reports

Conversation

@cv

@cv cv commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #12877; the parts still open after #12938.

Failure

  • Each platform's lifecycle suite runs in two partitions, and each partition reports only its own tests, so no report shows a platform's whole lifecycle suite.
  • Live / Docker and Live / Kind had no budgets.
  • cargo ci live-docker runs four nextest commands (the shared tests, then each gateway test with its own document). Each command rewrote live-docker.xml, so the step's timing report and its uploaded artifact showed only the last command's single test. In the 11 most recent v1 runs, the artifact reports 3–7 s; the four commands took 210–255 s together.

Decision

  • nemoclaw-build lifecycle-timing DIR --platform P --shards '[1, 2]' reads the downloaded lifecycle-P-SHARD-ATTEMPT artifacts. It uses each shard's latest attempt, names any shard without a report, and prints one report: a partition table, then the existing module and slowest-test tables over all tests. Partitions run in parallel, so the wall time is the slowest partition's.
  • A new Lifecycle timing job in native-platform.yml runs that command after the lifecycle shards and writes the report to its job summary. It runs on the platform's runner and takes the build tool from the existing lifecycle-inputs artifact, so it compiles nothing. It is continue-on-error, so a reporting problem cannot fail Test / PLATFORM. I put it here instead of the Test / PLATFORM gate job in rust.yml: the gate runs on Linux and has no build tool for the other platforms. Using it would mean compiling the tool inside a required check that has a 5-minute timeout.
  • Each partition keeps reporting and enforcing its own budget; the combined report enforces nothing.
  • The live-docker step now saves each nextest command's JUnit report and joins them (ci::timing::join_consecutive, run time summed), whether or not the tests pass.
  • .config/test-budgets.yaml adds budgets for the two live suites. Since ci: run the live suites in CI / Native with its bundle #12962, they run as the Live / Docker and Live / Kind jobs in native-platform.yml, still through cargo ci live-docker and cargo ci live-kind. cargo ci already enforces a budget for every step it finds in the file, on Linux CI runners. A new test checks that every step that writes a JUnit report has a budget.
    • live-kind: 270 s wall, 120 s per test. In 11 v1 runs: at most 199 s wall on linux_amd64 and 171 s on linux_arm64; slowest test 83 s.
    • live-docker: 360 s wall, 180 s per test. In the same v1 runs, the four commands together took at most 255 s on linux_amd64 and 210 s on linux_arm64. The v1 JUnit kept only the last command's test. This PR's run reports all 10 tests: 222 s wall on linux_amd64 and 197 s on linux_arm64. Its slowest test, retained_runtime_rebuilds_offline_without_protobuf_or_a_dependency_cache, took 110 s and 101 s. In v1 logs, that test ran past 120 s in 5 of 11 linux_amd64 runs. Subtracting the other six tests in its command (about 70 s here) from that command's v1 wall time (at most 218 s) puts it under about 150 s. That figure is an estimate, not a measurement.
  • docs/contributing/testing.md describes the combined report and lists the budgets, including the 120 s lifecycle limit from test: split the slow remote-service lifecycle scenarios #12965.

Validation

  • New tests in crates/nemoclaw-build/tests/ci.rs: joining consecutive JUnit reports, and the partitioned report (latest attempt, other platforms ignored, missing shard named, parse errors naming the artifact). The budget-coverage test failed on live-docker has no budget before the budgets were added.
  • cargo test -p nemoclaw-build --no-default-features --test integration -- ci::, cargo fmt --all --check, and cargo clippy --workspace --all-targets -- -D warnings pass.
  • Ran lifecycle-timing against v1 run 38046386392's lifecycle-linux_amd64-* artifacts: 121 tests, 249.5 s wall (partitions 249.5 s and 240.0 s), 1657.5 s summed.
  • CI on 55ccf01 passed, including all four Lifecycle timing jobs (about 11 s each on Linux). The linux_amd64 report shows 121 tests and 244.4 s wall. The live-docker artifacts now hold all 10 tests.
  • CI on ff88cbf, after merging v1, passed. The windows_amd64 report covers its three partitions: 112 tests, 198.8 s wall. The live jobs report "Within budget": live-docker 10 tests in 208.9 s on both Linux platforms, and live-kind 175.2 s on linux_amd64 and 158.0 s on linux_arm64. Those jobs use continue-on-error, so I checked the reports in their logs, not just the green checks.

…suites

Each lifecycle partition reported only its own tests. A Lifecycle timing
job in each platform's workflow now joins the partitions' uploaded JUnit
reports into one report in its job summary, from each shard's latest
attempt. Partitions keep enforcing their own budgets.

The live-docker step runs four nextest commands, and each overwrote the
previous JUnit report, so its report and artifact showed only the last
command's test. The step now joins the four reports. With every live test
reported, .config/test-budgets.yaml budgets live-docker and live-kind,
enforced on Linux like the other steps.

Refs #12877
@cv cv added area: ci CI workflows, checks, release automation, or GitHub Actions v1 NemoClaw v1 branch labels Oct 10, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

@cv
cv marked this pull request as ready for review October 10, 2026 12:36
@cv
cv merged commit 28a1872 into v1 Oct 10, 2026
48 of 49 checks passed
@cv
cv deleted the ci/combined-timing-reports branch October 10, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant