feat(pytorch): add PyTorch 2.13 currency scaffold#6365
Open
bhanutejagk wants to merge 8 commits into
Open
Conversation
Add DLC image configs, workflow callers, and release schedule entries
for PyTorch 2.13.0 on cu130 (Amazon Linux 2023). Covers ec2 and
sagemaker variants across cuda and cpu devices.
- docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock} pinned to
torch 2.13.0, torchvision 0.28.0, torchaudio 2.11.0 (held; upstream
audio has no 2.12/2.13 tag yet)
- 4 image configs and 4 workflow callers (pr + autorelease x ec2/sagemaker)
- autorelease crons: 30 19 UTC ec2, 00 23 UTC sagemaker on Tue/Thu
Align 2.13 currency image with newer upstream releases that were
available when scaffolding but held back from the initial 2.12-copy.
- nccl 2.26.2 -> 2.30.7-1: matches torch 2.13's bundled NCCL pin
(torch 2.13 requires source builds to use NCCL >= 2.23 and pins its
bundled runtime NCCL to 2.30)
- deepspeed 0.18.8 -> 0.19.2: 0.19.x train carries the torch >= 2.11
compatibility patches; 0.18.8 predates them
- transformer_engine 2.12.0 -> 2.16.1: security patch + cuDNN 9.18.1
requirement in the latest stable train (2.12 is 4 months old)
- efa 1.47.0 -> 1.49.0: latest EFA installer (libfabric 2.4.0amzn5.0,
efa driver 3.1.0, OFI NCCL Plugin 1.20.0)
- gdrcopy 2.4.4 -> 2.6: latest stable; new optional DMA-BUF backend
requires cuda driver 13.3+ but falls back to gdrdrv (current path)
- pyproject deepspeed==0.18.8 -> 0.19.2 mirrors the config bump
- uv.lock regenerated in 2.13/{cuda,cpu}
cryptography>=49.0.0 bump was attempted but reverted: mlflow>=3.9.0
caps cryptography<49 in the sagemaker extra. Revisit when mlflow's
upper cap loosens.
CPU configs unchanged (no gpu-only pins to bump).
The wheel-cache upload path in scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs `docker buildx build --target wheel-export`, which re-executes the entire Dockerfile to reach a scratch stage that COPYs the built wheels out of the flash-attn and TE builder stages. With MAX_JOBS=8, the concurrent flash-attn 2.8.3 + transformer-engine 2.16.1 source builds during that re-execution exceed the CI runner cgroup memory limit and get OOM-killed (SIGKILL / exit code 137, PR #6365 attempt 2). Halving MAX_JOBS to 4 roughly halves peak compile RAM, keeping the re-execution under the runner ceiling at the cost of a slower flash-attn / TE compile step. CPU configs unaffected.
bhanutejagk
force-pushed
the
pytorch-2.13-currency
branch
from
51e3d4c to
eb84c58
Compare
added 5 commits
July 9, 2026 13:41
Revert the temporary max_jobs=8->4 workaround (eb84c58). Root cause of the wheel-export OOM is x86-build-runner is CPU-sized and can't handle the concurrent flash-attn + TE source recompile that `docker buildx build --target wheel-export` triggers. Route pytorch GPU builds (device_type=gpu) to x86-vllm-build-runner, the existing GPU-compilation fleet already used by vllm and vllm-omni. CPU builds stay on x86-build-runner. No new CDK / infra changes needed; the fleet is already provisioned with GPU-compilation-appropriate RAM.
The dedicated x86-pytorch-build-runner fleet is defined in DLContainersInfraCDK mainline (commit 85a06e065, rebalanced by f4bcbc9c on 2026-06-30) - identical hardware to x86-vllm-build-runner and x86-sglang-build-runner, but reserved for pytorch. Route all pytorch builds (both CPU and GPU) to the dedicated fleet unconditionally, matching the vllm.pipeline.yml and sglang.pipeline.yml pattern. This replaces the temporary conditional that routed GPU builds to x86-vllm-build-runner as a stopgap. Cache locality on a dedicated fleet is expected to fix the wheel-export OOM: BuildKit finds the primary build's flash-attn + TE layers still cached on the same host, so the wheel-export re-run reduces to a COPY --from=<cached-layer> instead of a concurrent recompile. - pytorch.pipeline.yml: unconditional fleet:x86-pytorch-build-runner - ci-architecture.md: document the new fleet row
PR-6365 ECR vulnerability scan flagged 2 HIGH CVEs on 2.13 CPU images (also on SageMaker CPU variant). Both are unbounded-memory / catastrophic-backtracking bugs in soupsieve's CSS selector parser, fixed upstream in 2.8.4. soupsieve is transitive via beautifulsoup4 -> python-fasthtml -> fastai. Add an explicit direct pin `soupsieve>=2.8.4` to both cpu and cuda pyproject.toml (following prior transitive-CVE bump convention: msgpack, tornado, cryptography, aiohttp), regenerate uv.lock via `uv lock`. CVEs cleared: - GHSA-2wc2-fm75-p42x (unbounded memory on large selector lists) - GHSA-836r-79rf-4m37 (regex catastrophic backtracking on attribute selectors) Upstream fix pin: soupsieve>=2.8.4
The wheel-cache upload path re-runs `docker buildx build --target wheel-export` to extract compiled wheels from the wheel-export stage. Because that re-invocation passed NO --build-arg while the primary build passes ~15 of them, BuildKit computed a different cache key for every stage and had to recompile flash-attn + TE from source concurrently, OOM-killing the CodeBuild runner (exit 137 at upload_wheels.sh:45). Forward the same --build-args the primary build uses (read from EXTRA_BUILD_ARGS env var, the space-separated list of arg names the build-image action already computes via resolve_build_args.py). The wheel-export stage now hits BuildKit's local cache for every prior stage and reduces to a COPY --from=<cached-layer>, avoiding the recompile entirely.
Reverts commit 2705344. That change forwarded primary-build --build-args to the wheel-export re-invocation to make BuildKit cache-hit and avoid recompiling flash-attn + TE. No longer needed: the x86-pytorch-build-runner fleet has been bumped from m6a.8xlarge (128 GB) to m6a.12xlarge (192 GB) via DLContainersInfraCDK CR CR-288126860. The runner now has enough memory headroom to complete the cold-cache wheel-export recompile without OOM. Reverting keeps upload_wheels.sh in its upstream shape for other frameworks that share the pattern (though currently only pytorch_runtime uses this exact copy).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scaffold for the PyTorch 2.13 currency release on the V2 pipeline. Adds DLC image configs, workflow callers, per-version pyproject/uv locks, and release-schedule entries for PyTorch 2.13.0 on cu130 (Amazon Linux 2023), covering
ec2andsagemakercustomer types acrosscudaandcpudevice types.No shared Dockerfile changes —
docker/pytorch/Dockerfile.{cuda,cpu}remain shared across all versions; per-version pins live underdocker/pytorch/2.13/{cuda,cpu}/.Version pins
2.13.00.28.02.11.0(held)pytorch/audiohas no 2.12 or 2.13 release tag; latest is v2.11.0. Matches the 2.12 DLC pin.13.0.2(cu130)download.pytorch.org/whl/*with a complete torch + torchvision + torchaudio wheel set for the pinned versions (cu132 has torch+vision but no torchaudio).3.12uv lockre-resolved cleanly in both2.13/cudaand2.13/cpu.Files added
docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock}(4 files).github/config/image/pytorch/2.13-{ec2,sagemaker}-{cuda,cpu}.yml(4 configs).github/workflows/pytorch.{pr-2.13-cuda,pr-2.13-cpu,autorelease-2.13-ec2,autorelease-2.13-sagemaker}.yml(4 callers).github/release-schedule.yml— 2 new entries (Tue/Thu, 30-min grid,lightGPU class)Autorelease cron slots
Placed on Tue/Thu inside the documented 16:00–23:00 UTC release window, on the existing 00/30-minute grid, with a free 30-min buffer around each pick:
pytorch.autorelease-2.13-ec2.yml30 19 * * 2,4pytorch.autorelease-2.13-sagemaker.yml00 23 * * 2,4Both entries in
release-schedule.ymlmatch theon: schedule:blocks inside the workflow files (_prcheck.release-schedule.ymlshould pass).Test plan
pytorch.pr-2.13-cuda.ymltriggers on this PR (paths match.github/config/image/pytorch/2.13-*-cuda.yml,docker/pytorch/2.13/cuda/**) — build + sanity + security + single-gpu + efa + telemetry + unit + sagemakerpytorch.pr-2.13-cpu.ymltriggers on this PR — build + sanity + security + telemetry + unit_prcheck.release-schedule.ymlvalidates the cron correspondence)schedule+workflow_dispatchonly); they will fire on the first Tue/Thu after mergeworkflow_dispatchsmoke ofpytorch.autorelease-2.13-ec2.ymlafter merge to confirm the release path