ci: windows-directml integration job on self-hosted Windows GPU runners (#292) - #296
Merged
Merged
Conversation
…unners (#292) Runs RoFormer (incl. the exact model from the issue), MDX, and VR separations with --use_directml on the gha-runner-gpu-windows ephemeral fleet (T4, WDDM). Asserts per model: - DirectML actually engaged (torch device + DmlExecutionProvider) - reference-image quality (same thresholds as the CUDA jobs) - non-silent, finite output (RMS floor — silent garbage must fail) - RoFormer loads via the NEW implementation with no legacy fallback continue-on-error until stabilized, then promoted to the gate + ruleset. torch-directml is layered via pip in-job: current releases pin torch==2.4.1, so poetry's lock resolves the dml extra to torch-directml 0.1.13 (torch-1.13 era) — unusable. pip resolves the modern wheel in this job's venv only; no other platform's torch changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stall First live run failed with [WinError 206]: the SYSTEM-profile pypoetry venv path plus onnx's long test-data paths exceeds 260 chars. Use a short venv base (C:\venvs) and enable the LongPathsEnabled registry key (the runner executes as SYSTEM, and Python is longPathAware). Run also PROVED the stack: Windows ephemeral runner dispatched, booted, registered, ran the job; nvidia-smi reports '596.36, Tesla T4, WDDM'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
torch (c10.dll) and onnxruntime both fail with [WinError 126] on a fresh Windows Server image — they need the MSVC 2015-2022 runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
beveradb
added a commit
to nomadkaraoke/karaoke-gen
that referenced
this pull request
Jul 20, 2026
torch (`c10.dll`) and onnxruntime fail with `[WinError 126]` on the fresh Windows Server image — they need the MSVC 2015-2022 runtime. The `windows-directml` job installs it per-run as of nomadkaraoke/python-audio-separator#296; this bakes it in (with the standard 0/1638/3010 exit-code tolerance) so the image is correct by default. Picked up at the next bake. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…led separation Real-DML CI (first windows-directml runs) confirmed the last predicted gap: F.scaled_dot_product_attention is unimplemented on torch-directml and fails mid-forward with D3D12 'The parameter is incorrect.'. Attend now routes privateuseone tensors to the existing einsum math path, gated so every other device keeps flash behavior (equivalence test: math path matches SDPA output on CPU). MDX and VR already passed on real DirectML in the same run. The failure also exposed silent-failure bugs: Separator.separate() swallows per-file exceptions without tracebacks, and the CLI then printed 'Separation complete!' and exited 0 having produced nothing. Errors now log with exc_info, and the CLI exits 1 when no output files were produced. Bumps to 0.44.5 (code changes ship in the wheel). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-DML CI traceback (enabled by the previous commit's exc_info fix)
pinpointed the RoFormer forward failure: rotary_embedding_torch's
apply_rotary_emb concatenates empty unrotated edge slices, and
torch-directml rejects zero-sized tensor ops ('The parameter is
incorrect.'). These models rotate the full head dim, so the concat is a
mathematical no-op — compute t*cos + rotate_half(t)*sin directly on DML.
Gated on privateuseone; unit test proves exact equivalence with the
library implementation, and the forced-DML full-model equivalence tests
now exercise this path too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-DML run confirmed the new implementation loads (loading stats: new_implementation_success=1) and the rotary fix works, but inference then died with 'DML allocator out of memory': the einsum attention path materializes the full (b h i j) similarity tensor — ~1.3GB per layer at segment 801 across 62 bands — which torch-directml's allocator can't handle. Slice the batch dim (step 8) on DML only; equivalence test covers the ragged final slice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…L matrix The OOM persisted with batch slicing because torch-directml lowers einsum naively (broadcast multiply + reduce): the b×h×i×j×d intermediate is tens of GB at segment 801. torch.matmul maps to a real GEMM. Keep the batch slicing to bound the materialized similarity tensor. Also add MDX23C-8KFFT-InstVoc_HQ.ckpt (plain MDXC / TFC_TDF arch) to the DML matrix — its STFT wrapper already CPU-hops non-cuda/cpu devices, so it should pass unchanged; audibility checks only (no committed reference images), auto-downloaded on first run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…801 fp32) MDX23C (plain conv TFC_TDF) also OOM'd, proving the limit is torch-directml's allocator rather than our attention path. Run ckpt models at segment 256 — the documented resource knob for constrained GPUs. MDX and VR pass at defaults (0.93 waveform similarity). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-chunk memory now fits (segment 256 completed 2 chunks; MDX23C got to 3/35) but torch-directml's allocator doesn't reuse freed blocks across iterations, so long chunk loops still die with 'DML allocator out of memory'. Call torch_directml.empty_cache() (hasattr-guarded; added in its 0.2.x line) plus a gc pass after each batch, gated on privateuseone. No-op on every other device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion) Nine real-hardware CI iterations fixed every code-level DirectML incompatibility in the RoFormer/MDXC path (state-dict loading, complex ops, SDPA, rotary embedding, einsum lowering) — inference now runs on DML — but torch-directml's allocator cannot sustain the chunked inference loops of ckpt-based models: 'DML allocator out of memory' after a few chunks regardless of segment size, batch slicing, matmul lowering, gc, or empty_cache (verified on a 16GB T4; the conv-only MDX23C hits it too, proving it's not our attention code). torch-directml is effectively unmaintained upstream (last release Sept 2024). Until upstream fixes the allocator: - MDXC-family models automatically run on CPU under DirectML, with a clear warning; MDX and VR remain GPU-accelerated (CI-verified 0.93 waveform similarity) - AUDIO_SEPARATOR_FORCE_DML_MDXC=1 attempts DML anyway (for testing on other GPUs — AMD iGPUs with large shared memory may behave better) - windows-directml CI asserts the fallback fires AND output is correct vs references; MDX23C (plain TFC_TDF MDXC) added to the matrix - README DirectML table updated to the CI-enforced truth Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…DML job timeout
The roformer_loader module logger doesn't propagate to the CLI handler,
so 'with new implementation' never appears in CLI output — assert on the
separator's "Roformer loading stats: {'new_implementation_success': 1"
line instead. Separations themselves passed (3/5 green; the 2 roformer
failures were this assertion only). CPU-fallback inference takes ~30 min
of the job — raise timeout to 60.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The first consumer of the new
gha-runner-gpu-windowsephemeral fleet (karaoke-gen#881/#882, image baked today) — and the real-hardware verification of the #295 DirectML fixes.New
windows-directmljob (runs-on: [self-hosted, windows, gpu],continue-on-erroruntil stabilized) +tests/integration/test_windows_directml.py:--use_directmlthrough BSRoformer (model_bs_roformer_ep_317_sdr_12.9755.ckpt— the exact model from issue Windows DirectML testing results: MDX and VR work, RoFormer and Demucs currently fail for different backend limitations #292), MelBandRoformer, MDX, and VRDependency note: torch-directml is layered via
pipin the job, not the poetrydmlextra — modern torch-directml pinstorch==2.4.1, so poetry's lock resolved the extra to torch-directml 0.1.13 (Dec 2022, torch-1.13 era), which must not be used. pip resolves the modern wheel in this job's venv only; every other platform's locked torch is untouched.The job runs on this PR itself — first-ever dispatch of a Windows ephemeral runner.
@coderabbitai ignore
🤖 Generated with Claude Code