Repository navigation
ci: report each platform's whole lifecycle suite and budget the live suites - #12964
Merged
Merged
Conversation
…suites Each lifecycle partition reported only its own tests. A Lifecycle timing job in each platform's workflow now joins the partitions' uploaded JUnit reports into one report in its job summary, from each shard's latest attempt. Partitions keep enforcing their own budgets. The live-docker step runs four nextest commands, and each overwrote the previous JUnit report, so its report and artifact showed only the last command's test. The step now joins the four reports. With every live test reported, .config/test-budgets.yaml budgets live-docker and live-kind, enforced on Linux like the other steps. Refs #12877
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
v1 documentation preview: https://nvidia-preview-nemoclaw-v1-pr-12964.docs.buildwithfern.com/nemoclaw |
# Conflicts: # .github/workflows/native-platform.yml # docs/contributing/testing.md
# Conflicts: # .config/test-budgets.yaml
cv
marked this pull request as ready for review
October 10, 2026 12:36
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #12877; the parts still open after #12938.
Failure
cargo ci live-dockerruns four nextest commands (the shared tests, then each gateway test with its own document). Each command rewrotelive-docker.xml, so the step's timing report and its uploaded artifact showed only the last command's single test. In the 11 most recent v1 runs, the artifact reports 3–7 s; the four commands took 210–255 s together.Decision
nemoclaw-build lifecycle-timing DIR --platform P --shards '[1, 2]'reads the downloadedlifecycle-P-SHARD-ATTEMPTartifacts. It uses each shard's latest attempt, names any shard without a report, and prints one report: a partition table, then the existing module and slowest-test tables over all tests. Partitions run in parallel, so the wall time is the slowest partition's.Lifecycle timingjob innative-platform.ymlruns that command after the lifecycle shards and writes the report to its job summary. It runs on the platform's runner and takes the build tool from the existing lifecycle-inputs artifact, so it compiles nothing. It iscontinue-on-error, so a reporting problem cannot failTest / PLATFORM. I put it here instead of theTest / PLATFORMgate job inrust.yml: the gate runs on Linux and has no build tool for the other platforms. Using it would mean compiling the tool inside a required check that has a 5-minute timeout.ci::timing::join_consecutive, run time summed), whether or not the tests pass..config/test-budgets.yamladds budgets for the two live suites. Since ci: run the live suites in CI / Native with its bundle #12962, they run as theLive / DockerandLive / Kindjobs innative-platform.yml, still throughcargo ci live-dockerandcargo ci live-kind.cargo cialready enforces a budget for every step it finds in the file, on Linux CI runners. A new test checks that every step that writes a JUnit report has a budget.live-kind: 270 s wall, 120 s per test. In 11 v1 runs: at most 199 s wall on linux_amd64 and 171 s on linux_arm64; slowest test 83 s.live-docker: 360 s wall, 180 s per test. In the same v1 runs, the four commands together took at most 255 s on linux_amd64 and 210 s on linux_arm64. The v1 JUnit kept only the last command's test. This PR's run reports all 10 tests: 222 s wall on linux_amd64 and 197 s on linux_arm64. Its slowest test,retained_runtime_rebuilds_offline_without_protobuf_or_a_dependency_cache, took 110 s and 101 s. In v1 logs, that test ran past 120 s in 5 of 11 linux_amd64 runs. Subtracting the other six tests in its command (about 70 s here) from that command's v1 wall time (at most 218 s) puts it under about 150 s. That figure is an estimate, not a measurement.docs/contributing/testing.mddescribes the combined report and lists the budgets, including the 120 s lifecycle limit from test: split the slow remote-service lifecycle scenarios #12965.Validation
crates/nemoclaw-build/tests/ci.rs: joining consecutive JUnit reports, and the partitioned report (latest attempt, other platforms ignored, missing shard named, parse errors naming the artifact). The budget-coverage test failed onlive-docker has no budgetbefore the budgets were added.cargo test -p nemoclaw-build --no-default-features --test integration -- ci::,cargo fmt --all --check, andcargo clippy --workspace --all-targets -- -D warningspass.lifecycle-timingagainst v1 run 38046386392'slifecycle-linux_amd64-*artifacts: 121 tests, 249.5 s wall (partitions 249.5 s and 240.0 s), 1657.5 s summed.Lifecycle timingjobs (about 11 s each on Linux). The linux_amd64 report shows 121 tests and 244.4 s wall. The live-docker artifacts now hold all 10 tests.continue-on-error, so I checked the reports in their logs, not just the green checks.