ci: accelerate Subtensor CI with R2 sccache and tiered Fireactions pools#2857
ci: accelerate Subtensor CI with R2 sccache and tiered Fireactions pools#2857UnArbosFive wants to merge 30 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🛡️ AI Review — Skeptic (security review)VERDICT: VULNERABLE VERY HIGH account-age scrutiny, mitigated by repository admin permission and matching authorship; no Gittensor association; stacked branch targets bittensor-core-exploration. Static analysis found a steady-state cache-writer credential exposure affecting same-repository pull requests. No trusted review-instruction files were modified. Findings
ConclusionSame-repository PR-controlled code receives R2 writer credentials and can poison compiler-cache objects consumed by later trusted builds. Writer secrets must never enter pull-request jobs. 🔍 AI Review — Auditor (domain review)VERDICT: 👍 Gittensor association UNKNOWN; the author is a new account with repository admin permission, so workflow routing received direct scrutiny. The implementation otherwise matches the substantive description, and overlapping PRs are upstream/dependent work rather than competing implementations. No runtime change or spec-version bump applies. Description discrepancy: the body says the PR remains draft, while current metadata reports it as ready for review. No build or test was run because the remaining issue is statically evident. Findings
Prior-comment reconciliation
ConclusionApprove after applying the runner-label correction so both release production-binary builds retain the documented 16-core lane. 📜 Previous run (superseded)
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build uses the general 8-core pool
The PR promises to reserve fireactions-turbo for production-binary builds, and the equivalent PR validation build does so, but the release workflow routes this CPU-heavy binary build to fireactions-turbo-8. That contradicts the stated allocation and can lengthen the release critical path. Keep only the image validation and publishing jobs on the 8-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR says production-binary builds retain the reserved 16-core lane, and the equivalent PR build correctly uses fireactions-turbo. This release job performs the same cross-compiled production build but routes both architectures to the general 8-core pool. Keep the expensive compilation on the documented reserved lane; the image and publish jobs can remain on fireactions-turbo-8.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR says production-binary builds retain the 16-core fireactions-turbo lane, but this production binary matrix is routed to the general 8-core pool. This is the cross-compilation-heavy build the capacity policy explicitly reserves for 16 cores.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that production-binary builds retain the reserved 16-core lane, but this release job routes both architecture builds to the general 8-core pool. This contradicts the documented capacity policy and can lengthen release builds or contend with general workloads. Route this compilation job to fireactions-turbo-16; the image-validation and publishing jobs can remain on the 8-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
UnArbosFive
changed the title
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
c157fb5 to
db0d556
Compare
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that both amd64 and arm64 production-binary builds retain 16 cores, but this release matrix still runs them on fireactions-turbo-8. This also diverges from the equivalent PR validation job in check-docker.yml. Route this compile-heavy job to the 16-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR states that both amd64 and arm64 production binary builds retain 16 cores, and the equivalent PR validation job uses fireactions-turbo-16. This release matrix still selects the 8-core pool, contradicting that allocation and potentially doubling latency on the release-critical builds.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description says the amd64 and arm64 production-binary builds retain the 16-core lane, but this release workflow routes both builds to the 8-core pool. This also differs from the equivalent PR validation build in check-docker.yml. Route this compile-heavy matrix to fireactions-turbo-16.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
This production-binary matrix still runs on the 8-core general lane, contrary to the PR’s stated allocation of the amd64 and arm64 production builds to 16 cores. It also differs from the equivalent matrix in check-docker.yml. Route this compile-heavy release job to the 16-core pool.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
163e63a to
c5dd58b
Compare
UnArbosFive
force-pushed
the
codex/fireactions-runner-density-experiment
branch
from
c5dd58b to
d981a4c
Compare
| binary: | ||
| name: production binary (${{ matrix.platform.arch }}) | ||
| needs: setup | ||
| runs-on: [self-hosted, fireactions-turbo-8] |
There was a problem hiding this comment.
[MEDIUM] Release production build still uses the general pool
The PR description explicitly assigns both amd64 and arm64 production-binary builds to the 16-core lane, but this release matrix still runs them on fireactions-turbo-8. This contradicts the documented capacity design and leaves the most expensive release compiles on half-sized runners. Route this build job to the dedicated 16-core pool, consistent with check-docker.yml.
| runs-on: [self-hosted, fireactions-turbo-8] | |
| runs-on: [self-hosted, fireactions-turbo-16] |
There was a problem hiding this comment.
Addressed as a documentation contradiction. The PR description now distinguishes the per-PR amd64 production build, which remains on fireactions-turbo-16, from the infrequent release-only amd64/arm64 matrix, which intentionally stays on fireactions-turbo-8 to favor fleet efficiency over release latency. No runner change is intended here.
|
🔄 AI review updated — Skeptic: SAFE Auditor: 👍 |
| environment: | ||
| name: ${{ github.event_name == 'workflow_dispatch' && 'sccache-reader' || 'sccache-writer' }} |
There was a problem hiding this comment.
[HIGH] PR-controlled jobs receive cache-writer credentials
This selects sccache-writer for every event except workflow_dispatch, including pull_request. The job then checks out and executes PR-controlled composite actions, build scripts, and compiler inputs while passing the writer secrets. A same-repository PR can exfiltrate these credentials or overwrite cache objects for later trusted builds; the event checks inside sccache-configure.sh occur only after the secrets have already entered PR-controlled code.
Use the reader environment for every pull-request job and do not pass writer secrets there. Apply the same correction to the other PR-triggered workflows introduced here.
| environment: | |
| name: ${{ github.event_name == 'workflow_dispatch' && 'sccache-reader' || 'sccache-writer' }} | |
| environment: | |
| name: ${{ github.event_name == 'push' && 'sccache-writer' || 'sccache-reader' }} |
|
🔄 AI review updated — Skeptic: VULNERABLE |
1 participant
Summary
This is the unified Subtensor CI acceleration change. It supersedes #2855 and combines:
sccachefor Rust compilationfireactions-turbo-16andfireactions-turbo-8runner lanesShared compiler cache
sccachecentrally through the shared Rust setup pathmainpushes, avoiding a second post-CI compilationdevnet/testnetupdates, Amsterdam-noonmainmaintenance, and restrictedmain/devnet/testnetmanual recoverysccache --show-statsfor cache observabilityWriter policy
The trust boundary is repository contents-write access, inferred from the PR source; it is not hard-coded to the
arbosteam.arbosmembers, direct collaborators, GitHub Apps, or other principals that can push a branch insideRaoFoundation/subtensorare therefore trusted equivalently. Organization members without repository write access are not.pull_requestmaindevnetortestnetmainat 12:00Europe/Amsterdammainfor an enumeratedmain/devnet/testnetsourceworkflow_dispatchsccache-readerenvironment and preserves manual CI behaviorpull_request_target, arbitrary push, malformed event, partial credentialsR2 writes are content-addressed. A changed source, compiler, feature set, flag set, or target creates a different key, so artifacts used by older PRs remain alongside newer results until the 30-day lifecycle removes them.
sccachestores each successful compiler invocation independently: if a later crate or test fails, earlier valid compilation results remain useful and require no rollback.The bucket-scoped writer credential is intentionally available to later steps and Rust build scripts in an authorized job because
sccacheitself needs it. Consequently, every repository-write principal is effectively trusted with that credential for the duration of their ephemeral job. Read-only credentials remain in Fireactions MMDS; writer credentials remain exclusively in the GitHubsccache-writerenvironment.The persistent
Benchmarkingrunner remains read-only so a background sccache process cannot retain the writer credential across jobs. The accepted containerizeddocker-localnetbuild remains outside sccache; its roughly 15-minute main/release build does not extend ordinary PR critical-path compilation.Measured CI impact
The final matched comparison uses #2846 at
729385das the legacy-runner/cache baseline and this PR ate9cf2dd. Both ran against the same Subtensor source on 2026-07-12, at 09:03 and 12:31 UTC respectively.These figures intentionally do not claim a 36% reduction in full merge latency. Try-runtime, clone-upgrade, and Zombienet execution are endpoint/test-bound and dominate the tail; they gain little from faster compilation hardware. Try-runtime was materially slower from BHS, which is why it remains on the European pool.
The one-pass writer policy removes the separate PR warming compile. Trusted jobs selected R2 writer mode with zero cache write errors, while parallel jobs immediately reused each other's outputs: the amd64 production binary reached 99.94% cache hits,
cargo test99.32%, all-feature clippy 92.90%, default clippy 54.58%, and cargo fix 64.95%. Average R2 writes were 0.17-1.03s; the production job had only two misses at the upper end.Runner-density design
The ten-machine 9950X experiment is intentionally fixed at:
fireactions-turbo-16fireactions-turbo-8The three PR-critical compile jobs retain 16 cores:
cargo testGeneral Rust checks, SDK/docs, E2E builds and execution, and Docker packaging use the 8-core pool.
The release-only production-binary matrix in
docker.ymlalso intentionally usesfireactions-turbo-8for both amd64 and arm64. It does not gate pull requests, runs infrequently, and favors fleet efficiency over minimum release-build latency. The per-PR amd64 production build remains onfireactions-turbo-16because it is on the review critical path. This distinction is deliberate; the release matrix is not expected to mirrorcheck-docker.yml's runner assignment.Try-runtime uses the four interchangeable
fireactions-tryruntimerunners on AX102. Historical AX102 runs were roughly twice as fast as BHS despite fewer cores, and direct endpoint probes showed materially lower archive-RPC latency. A separatefireactions-tryruntime-finneyfallback remains deployed for compatibility, but workflows in this PR do not select it.Observed capacity
The final run peaked at 3 of 8
fireactions-turbo-16runners and 15 of 24fireactions-turbo-8runners. One full PR is strictly queue-free.Two perfectly aligned PRs use 6/8 heavy runners but request 30/24 light runners, so up to six light jobs queue. The observed short-job completion profile implies roughly 1-3 minutes of light-lane delay and no heavy-lane delay. Three aligned PRs request 9/8 heavy and 45/24 light runners; one critical heavy job waits about 4:40 for the first slot and up to 21 light jobs queue, so three is outside the low-latency design point. Operationally this is a two-concurrent-PR fleet with modest bursting, not a two-PR queue-free fleet.
The external AX102 try-runtime pool remains a separate bottleneck: each PR requests three of its four runners, so a second overlapping PR queues two try-runtime jobs even when the 9950X fleet is healthy.
Infrastructure dependency
The R2 bucket, MMDS reader configuration, writer secrets, and matching Fireactions pools must remain deployed from the infrastructure repository. Excess compatibility capacity is intentionally excluded from the designed 8/24 runner totals so it can be retired without reducing the experiment pool.
Rollout status
sccache-writeris restricted tomain,devnet,testnet, and PR merge refs;sccache-readercontains no secrets and preserves feature-branch dispatches729385d); final CI fore9cf2ddis running normally