Repository navigation
refactor(ci3): consume ci3 as a shared submodule - #25516
Open
charlielye wants to merge 9 commits into
Open
charlielye wants to merge 9 commits into
charlielye wants to merge 9 commits into
Conversation
charlielye
marked this pull request as ready for review
September 21, 2026 11:48
charlielye
requested review from
iakovenkos,
ledwards2225 and
ludamad
as code owners
September 21, 2026 11:48
charlielye
force-pushed
the
cl/ci3-submodule
branch
from
October 7, 2026 16:16
a557c8d to
f3e2d68
Compare
charlielye
force-pushed
the
cl/ci3-submodule
branch
from
October 22, 2026 14:51
f3e2d68 to
9c9b128
Compare
ci3 now lives in aztec-labs-eng/ci3, shared with aztec-node, so the two repos stop drifting: this repo's copy was months behind on the logic both share. The tree here is replaced by the submodule. Scripts specific to this repo (the merge-train and canary notifiers, the public release assertion, the GCP artifact login and the npm release publisher) move to ci3-local/ rather than into the shared repo, carrying their latest changes, with call sites updated. ci.sh and bootstrap.sh check the submodule is initialised before sourcing it. ci.sh and the two barretenberg benchmark scripts stop re-sourcing source_redis, source_refname and source_cache, which ci3/source already includes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workflows can use ci3 without naming it, through any repo script that sources it, so every checkout of this repository gets it, via a step that initialises only ci3: `submodules: true` would also clone noir-repo and the contract libraries into every job. The step is a no-op on refs from before ci3 became a submodule, which the sync and port workflows check out. The two workflows calling the scripts now in ci3-local/ use the new paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci3's scripts used only by ci3 itself now live under internal/ (still on PATH), including source_redis, source_refname and source_cache, which this branch already stops sourcing directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes the benchmark run, which failed before starting: sem_sched ran its semc helper by a path relative to its own directory, which moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared ci3 retries a failing test only when a matching entry sets retry: true; owners no longer grant a retry. Every entry with owners sets it, so the same tests stay tolerated as flakes. The header documents retry, notify_on and channel.
Runner logs published to S3, one log_publish, no SSH tunnel to the CI redis, the dashboard's server API removed, bench_upload, explicit test retries, and CURRENT_VERSION dropped.
charlielye
force-pushed
the
cl/ci3-submodule
branch
from
November 5, 2026 17:37
4e1a1f6 to
bc44005
Compare
Picks up aztec-labs-eng/ci3#9: a failed docker compose up fails the test, carried over from this repo's in-tree copy (#25691).
ci3-local held what was left of this repo's in-tree ci3 that the shared ci3 does not carry. The scripts this repo uses move to scripts/: assert_public_release, gcp_artifact_login, publish_npm_release, and merge_train_failure_slack_notify as scripts/merge-train/slack-notify.sh beside the other merge-train scripts (without its merge-queue dequeued mode). Removed with no callers: merge_train_stale_check (its workflow is gone), release_canary_slack_notify (hardcoded to a canary branch and only called from the old bootstrap_ec2) and npm/.npmrc (deploy_npm uses source_npm_auth). The shared bootstrap_ec2 does not post merge-train failures, so ci3.yml does: a failed CI run on a merge-train PR posts its build log to the train's channel.
next has no merge queue, so the dequeued job never runs. The workflow keeps its other job, the post when a merge-train PR merges, and is renamed after it: merge-train-notify.yml.
Collaborator
Flakey Tests🤖 says: This CI run detected 2 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces this repo's in-tree
ci3/with the shared repo aztec-labs-eng/ci3, which was split out ofaztec-node's copy with its full history (531 commits).Why
The two copies had drifted: 43 files differed, and this repo's version was months behind on the logic both share. Examples of what this repo picks up:
cache_content_hash: ordered include/exclude patterns, where!-negation and last-match-wins are supported.docker_isolate: the container-name reclaim race fix (a same-name sibling is waited out instead of killed mid-run).source_refname:ci_default_branch/ci_dashboard_section, replacing a hardcodedmaster.deploy_npm: per-channel npm dist-tags instead of everything sharingprerelease.release_prep_package_json:--checkvalidation and scoped-resolutions mode.log_ci_run: dashboard sections scoped by<org>/<repo>, which is what makes this repo's runs show up on the dashboard landing page at all — today they land in the legacy unscopedci-run-prskey and are invisible there.What is in this PR
ci3/becomes a submodule pinned tomain, declaredshallow = true.scripts/:assert_public_release,gcp_artifact_login,publish_npm_release, andmerge_train_failure_slack_notifyasscripts/merge-train/slack-notify.sh. Call sites are updated.publish_npm_releasemoves into the shared ci3 with feat: pack mode for deploy_npm and a trusted-publishing publisher aztec-labs-eng/ci3#2. Removed because nothing calls them:merge_train_stale_check,release_canary_slack_notify(hardcoded to a canary branch) andnpm/.npmrc.merge-queue-dequeue-notify.ymlloses its dequeued job (there is no merge queue) and becomesmerge-train-notify.yml, which keeps the post when a merge-train PR merges.ci.shandbootstrap.shcheck the submodule is initialised and say how to fix it, since both source ci3 on their first line.bootstrap.shstill owns pulling the remaining submodules.submodules: trueis deliberately not used: here it would also clone noir-repo and the contract libraries into every job. The step is a no-op on refs from before ci3 became a submodule, which the sync and port workflows check out.ci.shand the two barretenberg benchmark scripts stop re-sourcingsource_redis,source_refnameandsource_cache, whichci3/sourcealready includes.bootstrap_ec2initialises the ci3 submodule on the remote EC2 instance (its shallow fetch does not bring submodules) and derives the checkout directory fromGITHUB_REPOSITORYrather than hardcoding a repo name.Known deltas to work through
This is a draft because adopting the shared version changes behaviour that was repo-specific here:
bootstrap_ec2. The old copy calledmerge_train_failure_slack_notifyandrelease_canary_slack_notifyfrombootstrap_ec2; the shared version has no such calls. The scripts still exist inci3-local/and themerge-queue-dequeue-notify.ymlpath still works, but these two call sites are currently dropped. Resolved: the canary notifier had no live use and is removed. Merge-train failures now post from aNotify merge-train failurestep inci3.yml, which links the build log (CI_LOG_ID).run_test_cmd. The shared version tracks history and notifies flakes formain-branch runs; this repo's keyed offgh-readonly-queue/,backport-to-v2-stagingand the spartan trains. Needs a config knob rather than a hardcoded branch list.mastervs the default branch.CI_FULLnow comes fromci_default_branch. Checked: it reads the remote's default branch, sonexthere.bootstrap_ec2exportsGCP_HELM_SA_KEY_B64; this repo previously exportedGCP_PRIVATE_NPM_DEPLOY_B64. Checked: onlybootstrap_ec2uses the name, to pass the key from the runner to the instance, so the rename is self-contained.deploy_npmsourcessource_npm_auth; this repo usedNPM_CONFIG_GLOBALCONFIGwithci3/npm/.npmrc. That file was unreferenced and is removed. To confirm with aci-release-prrun.bootstrap_ec2does not do that. Likely covered:.github/ci3.shalready setsINSTANCE_POSTFIXto the PR's commit count for merge-train PRs, which separates their instances.Each is a candidate for a per-repo config file in the shared repo rather than a fork of the script.
Depends on
The matching
aztec-nodePR, which is the pilot consumer and will land first.Rebuilt on next (latest: 2026-11-05)
Rebased onto current
next, andci3pinned to ci3main(a379ffc).ci3/run_compose_test: a faileddocker compose upnow fails the test. The in-tree copy is dropped here; the change was ported to the shared ci3 in fix: a failed docker compose up fails run_compose_test aztec-labs-eng/ci3#9, which the pin includes..test_patterns.ymlentry setsretry: true; owners no longer grant a retry. All 43 entries with owners now set it, so the same tests stay tolerated as flakes. aztec-node made the same change in its bump.log_publishfor every log write;bench_upload;CURRENT_VERSIONdropped (unused here: only the docs version scripts have a local variable of that name);./bootstrap.sh init.labs-patchespatch 0020 not applying to the labs pin.nexthas since fixed that (e63137d).Earlier rebuild (2026-10-22)
Rebuilt on current
nextwith the same net effect, pinned to ci3b505e87, which carriesaztec-node'sci3/changes since the split. Changes to this repo'sci3/since the original base, and where they went:ci3-local/. Itsrun_test_cmdflake-list edit applies to policy this repo no longer controls; see delta 2 above. The sharedslack_notify_with_claudebox_kickoffstays, sinceaztec-nodestill uses it.publish_npm_releaseaccepts@aztec-labs): carried intoci3-local/.devbox:3.2/build:3.2images, and pins this repo's rebuilt AMIs, so this repo's AMIs carry over unchanged. chore: bump Foundry to 1.8.5 #25602'sami_update.shfix for the neweraws_request_instanceinterface is adopted there too.ipc-runtime/bootstrap.shhas since started callingassert_public_release, which is updated to theci3-local/path along with the other callers.Also fixed in an earlier rebuild:
merge-queue-dequeue-notify.ymlrunsci3-local/merge_train_failure_slack_notify, which callsci3/slack_notify, but its checkouts were not getting the submodule, so dequeue notifications would have failed.nextno longer has a merge queue (the lastmerge_grouprun was 2026-09-17), so deltas 1 and 2 above matter less than when they were written: they describe merge-queue and merge-train behaviour.