Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions .claude/skills/merge-train-infra/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The merge-train system is fully automated via GitHub Actions in `.github/workflo

5. **Recreation & Wakeup** (`merge-train-recreate.yml`): Triggered when a PR is closed (merged). If the merged PR's head branch starts with `merge-train/`, recreates the branch from the base branch (usually `next`). Then runs `scripts/merge-train/wakeup-prs.sh` to add the `ci-wakeup-pr-after-merge` label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in `ci3.yml` so it can be re-applied on subsequent merges.

6. **Failure Notification** (`merge-queue-dequeue-notify.yml`): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with `merge-train/` and the PR was NOT merged, sends a Slack notification via `ci3/merge_train_failure_slack_notify`.
6. **Slack Notifications** (`scripts/merge-train/slack-notify.sh`, which maps each train branch to its team channel): a failed CI run on a merge-train PR posts the build log from the `Notify merge-train failure` step in `ci3.yml`, and a merged merge-train PR posts from `merge-train-notify.yml`.

## Release-Line Ports

Expand Down Expand Up @@ -69,16 +69,16 @@ if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET
fi
```

### Failure Notification (`ci3/bootstrap_ec2`)
### Failure Notification (`.github/workflows/ci3.yml`)

When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_notify` to send failure notifications to the appropriate Slack channel based on the branch name.
When the CI run of a merge-train PR fails, the `Notify merge-train failure` step calls `scripts/merge-train/slack-notify.sh` with the build log id (`CI_LOG_ID`, set by `ci3/bootstrap_ec2`), which posts to the train's Slack channel.

## Creating a New Merge Train

1. Create a branch from the desired base (`next` for most trains; a release line like `v5-next` for a release-specific train) with naming pattern `merge-train/{team}`. For a v5-release train use the `-v5` suffix (`merge-train/{team}-v5`): `merge-train-create-pr.yml` routes any `*-v5` branch to a `v5-next` base automatically and adds the `private-port-next` label.
2. Add the branch to the loop in `.github/workflows/merge-train-next-to-branches.yml`. If it tracks a base branch other than `next`, also add that base to the workflow's `push` trigger and pass it as the second argument to `merge-next.sh` (see the `v5-next` → `-v5` trains wiring)
3. For a base other than `next` that the `*-v5` convention does not already cover, set the PR base in `.github/workflows/merge-train-create-pr.yml`. Either way, add a stale-check job that passes `BASE_BRANCH` in `.github/workflows/merge-train-stale-check.yml`
4. Add the branch-to-Slack-channel mapping in `ci3/merge_train_failure_slack_notify`
3. For a base other than `next` that the `*-v5` convention does not already cover, set the PR base in `.github/workflows/merge-train-create-pr.yml`
4. Add the branch-to-Slack-channel mapping in `scripts/merge-train/slack-notify.sh`
5. Optionally add CI mode overrides in `.github/ci3_labels_to_env.sh` and `bootstrap.sh`
6. Push code to the branch -- automation handles PR creation from there

Expand All @@ -94,7 +94,7 @@ When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_noti
| `.github/workflows/merge-train-next-to-branches.yml` | Syncs `next` into all train branches; defines active branches |
| `.github/workflows/merge-train-recreate.yml` | Recreates branch after merge |
| `.github/workflows/merge-train-update-pr-body.yml` | Updates PR body with commit list (merge-train branches) |
| `.github/workflows/merge-queue-dequeue-notify.yml` | Slack notification on merge-queue dequeue |
| `.github/workflows/merge-train-notify.yml` | Slack notification when a merge-train PR merges |
| `.github/workflows/squashed-pr-check.yml` | Squash enforcement (skipped for `ci-no-squash`) |

### Scripts
Expand All @@ -106,16 +106,15 @@ When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_noti
| `scripts/merge-train/update-pr-body.sh` | Updates PR body with meaningful commits |
| `scripts/merge-train/squash-pr.sh` | Squashes PR commits (used by `ci-squash-and-merge` label) |
| `scripts/merge-train/wakeup-prs.sh` | Adds `ci-wakeup-pr-after-merge` label to qualifying PRs after branch recreation |
| `scripts/merge-train/slack-notify.sh` | Slack failure and merge notifications, with the branch-to-channel mapping |

### CI Configuration

| File | Purpose |
|---|---|
| `.github/ci3_labels_to_env.sh` | CI mode selection based on labels and target branches |
| `.github/ci3.sh` | Instance postfix for merge-train parallelism |
| `ci3/merge_train_failure_slack_notify` | Slack failure notification with branch-to-channel mapping |
| `ci3/run_test_cmd` | Test history tracking for merge-train branches |
| `ci3/bootstrap_ec2` | EC2 failure notification trigger |
| `bootstrap.sh` | CI mode definitions (`ci-barretenberg`, `ci-full`, etc.) |

### Other Scripts
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/auto-rebase-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ jobs:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
ref: ${{ github.event.pull_request.head.ref }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi


- name: Authenticate with GitHub CLI
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/barretenberg-nightly-debug-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ jobs:
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
ref: next
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/barretenberg-nightly-slow-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ jobs:
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
ref: next
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/ci3-external.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ jobs:
# The commit to checkout. We want our actual commit, and not the result of merging the PR to the target.
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

# NOTE: in ci3.yml we just rely on draft mode not being mergable.
# Here we are a little more careful than just skipping the worklfow, in case of an edge case allowing merge.
Expand Down
19 changes: 18 additions & 1 deletion .github/workflows/ci3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,10 @@ jobs:
# Persisted for authenticated reads (e.g. the aztec-packages-private mirror); pushes swap in
# a bot-PAT remote at each push site.
persist-credentials: true
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
Expand Down Expand Up @@ -135,6 +139,15 @@ jobs:
# NOTE: $CI_MODE is set in the Determine CI Mode step.
run: ./.github/ci3.sh $CI_MODE

# A merge-train failure is posted to the train's team channel; the script maps the branch to
# its channel and ignores the others. CI_LOG_ID is the build log, set by ci3/bootstrap_ec2.
- name: Notify merge-train failure
if: failure() && startsWith(github.event.pull_request.head.ref, 'merge-train/')
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
REF_NAME: ${{ github.event.pull_request.head.ref }}
run: ./scripts/merge-train/slack-notify.sh "${CI_LOG_ID:-}"

- name: Post-Actions
env:
SHOULD_SQUASH_MERGE: ${{ contains(github.event.pull_request.labels.*.name, 'ci-squash-and-merge') && '1' || '0' }}
Expand Down Expand Up @@ -173,6 +186,10 @@ jobs:
with:
fetch-depth: 1
persist-credentials: false
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
Expand Down Expand Up @@ -200,4 +217,4 @@ jobs:
S3_BUILD_CACHE_AWS_PARAMS="--region us-east-2" ./ci3/cache_download "npm-release-${GITHUB_REF_NAME}.tar.gz" "$dir"

- name: Publish
run: ./ci3/publish_npm_release "$NPM_RELEASE_DIR/npm-release"
run: ./scripts/publish_npm_release "$NPM_RELEASE_DIR/npm-release"
4 changes: 4 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
Expand Down
42 changes: 0 additions & 42 deletions .github/workflows/merge-queue-dequeue-notify.yml

This file was deleted.

4 changes: 4 additions & 0 deletions .github/workflows/merge-train-auto-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ jobs:
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Run auto-merge script (merge-train)
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/merge-train-create-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ jobs:
- uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
fetch-depth: 0
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Check if PR exists and create if needed
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/merge-train-next-to-branches.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ jobs:
with:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure Git
run: |
Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/merge-train-notify.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Notify Slack when a merge-train PR merges

on:
pull_request:
types: [closed]

jobs:
notify-merged:
name: Notify Slack (Merged)
runs-on: ubuntu-latest
if: github.event.action == 'closed' && github.event.pull_request.merged == true && startsWith(github.event.pull_request.head.ref, 'merge-train/')
steps:
- name: Checkout
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
ref: ${{ github.event.pull_request.base.ref }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Send Slack notification
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
REF_NAME: ${{ github.event.pull_request.head.ref }}
TARGET_BRANCH: ${{ github.event.pull_request.base.ref }}
PR_URL: ${{ github.event.pull_request.html_url }}
run: ./scripts/merge-train/slack-notify.sh --merged
4 changes: 4 additions & 0 deletions .github/workflows/merge-train-recreate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ jobs:
with:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure Git
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/merge-train-update-pr-body.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ jobs:
with:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Update merge train PR body
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/nightly-release-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ jobs:
with:
ref: ${{ matrix.branch }}
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Create Nightly Tag
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/public-next-pr-body.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ jobs:
ref: next
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Update holding PR body
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/public-v5-next-pr-body.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ jobs:
ref: v5-next
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Update holding PR body
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/redo-typo-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ jobs:
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Authenticate with GitHub CLI
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/socket-fix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ jobs:
with:
ref: next
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Run
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/sync-upstream-next.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ jobs:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
persist-credentials: true
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure Git
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/sync-upstream-v5-next.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ jobs:
fetch-depth: 0
token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
persist-credentials: true
- name: Check out the ci3 submodule
# Only the ci3 submodule: `submodules: true` would also clone noir-repo and the
# contract libraries. A no-op on refs from before ci3 became a submodule.
run: if [ "$(git ls-files -s ci3 | cut -c1-6)" = 160000 ]; then git submodule update --init --depth 1 ci3; fi

- name: Configure Git
run: |
Expand Down
5 changes: 5 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,8 @@
branch = main
shallow = true
update = none
[submodule "ci3"]
path = ci3
url = https://github.com/aztec-labs-eng/ci3.git
branch = main
shallow = true
Loading
Loading