feat(tts/zipvoice): LuxTTS/ZipVoice CoreML conversion + ANE-canonical decoder rewrite#75
Open
Alex-Wengg wants to merge 18 commits into
Open
feat(tts/zipvoice): LuxTTS/ZipVoice CoreML conversion + ANE-canonical decoder rewrite#75Alex-Wengg wants to merge 18 commits into
Alex-Wengg wants to merge 18 commits into
Conversation
…ill) TextEncoder (8.3MB) + FmDecoder (238MB) fp16 mlpackages with host-side duration expansion and 4-step anchor-Euler solver. Parity vs torch: text encoder cos 0.999997, per-step decoder cos >=0.9987, end-to-end log-mel cos 0.99925. M5 Pro: 14.1ms/step on GPU (~130x realtime core), ANE resident (98%) but slow (286ms/step) pending layout work. Workarounds documented in coreml/trials.md: frozen rel-pos encodings (convert_scaled_to_non_scaled jit-scripts encoder_pos), aten::Int on 1-element arrays, SimpleDownsample zero-size pad. Vocoder (dual-head 48k Vocos + Linkwitz-Riley crossover) stays in torch for this trial. Requested in FluidAudio#49.
benchmark.py: per-component + end-to-end latency across compute units. M5 Pro: GPU core 62ms (48x RT core, 21.5x e2e with torch vocoder, which is now the bottleneck at 77ms); CPU 4.2x; ANE 2.4x. torch-cpu decoder baseline 405ms/step vs CoreML GPU 14.2ms (28x). audio_similarity.py: level/energy/spectral comparison vs oracle wav. RMS delta -0.094dB, envelope corr 0.9954, all octave bands within +/-0.27dB, spectral convergence 0.154 - phase-only divergence.
whisper-base transcribes CoreML and pytorch outputs identically. Onset clipping traced to upstream speed*1.3 duration squeeze (reproduces in pure-torch oracle); speed 0.9-1.0 recovers the full sentence.
…nuation Not whisper (silence-pad test), not token alignment (two-segment expansion is transcript-identical). A prompt hard-cut mid-phrase makes the model elide sentence-initial words to continue seamlessly; a prompt ending at a natural pause synthesizes the full sentence. speed*1.3 amplifies the elision. Integration: VAD-boundary prompt trimming + speed default 1.0.
--max-tokens/--max-frames on the converter; synth() bucket-aware. 2048-frame/512-token variant: 10-14s utterances transcribe identically between CoreML and PyTorch with onsets intact. Decoder step 14.0ms@1024 -> 35.4ms@2048 (subquadratic via downsampled stacks).
6-bit palettization is the ship candidate: 101MB weights, full GPU speed (15.0ms/step), 419MB footprint (2.4x down from fp16), -0.45dB, identical transcript. int8 linear crashes MPSGraph on GPU (MLIR pass assertion, macOS 26.5) but works on ane/cpu. int4 per-tensor loses 2.1dB - needs grouped-channel LUT (iOS18) to be viable.
… bandwidth 99.5% ANE residency, 0.6ms of CPU fallback, quantization changes nothing: the 286ms/step is O(T^2) rel-pos attention tiling badly on ANE (superlinear bucket scaling vs near-linear GPU). Needs the ane-transformers treatment ((B,C,1,S) QKV, split softmax, no gather) as its own trial. 256-frame ANE buckets are usable today (~17x RT).
Module isolation shows every module pays the seq-first layout tax (ff 5x vs GPU, summed = the whole 286ms). Same-weights (B,C,1,S) conv2d recast of feedforward: 6.83 -> 0.50ms on ANE. Full rewrite projected 286 -> ~40-60ms/step: iPhone-viable ANE-first TTS.
=1024, 100% ANE, exact parity One full Zipformer2EncoderLayer rewritten to (1,C,1,S) conv2d form (coreml/ane/): Linear->1x1 conv, channel-axis BiasNorm/Bypass, per-head attention with last-axis softmax, rel->abs positional reindexing baked into a constant pos_abs buffer (broadcast-mul + reduce; the pad/reshape skew flatten exceeds ANE dim limits), depthwise k=31 split into k=15+15+1 (ANE grouped-conv width limit is 15) using conv padding + slices instead of standalone pads. Parity fp32 eager: whole layer max_abs 2.4e-6, cos 1.0; all submodules < 2e-6. CoreML fp16 ANE output vs torch fp32: cos 1.000000. Latency S=1024 (M5 Pro): ANE 35.9 -> 4.14 ms (8.7x), GPU 2.1 ms; 166/166 ops on ANE, zero CPU fallback (original: 2 dw convs on CPU). Template for the whole-decoder ANE rewrite.
…, 100% ANE, 25.5 MB footprint AneFmDecoder rewrites all 16 Zipformer layers plus decoder plumbing (in/out proj, t+guidance embedding, downsample/upsample/bypass, float padding mask) in (1,C,1,S) form, reusing the trial-2 layer submodules. pos_abs sharing: per-layer folded buffers would cost ~260 MB; instead an R=32 SVD basis of the per-seq-len posproj union (numerical rank ~27, reconstruction <=3.7e-8) gives ONE shared constant per distinct S (67+16.8+4.2 = 88 MB fp16, 3 buffers), with per-layer coefficients folded into the attention in_proj p block. Eager fp32 parity vs torch@1024+mask: max_abs<=3.8e-4, cos=1.00000000 all 4 solver steps. vs the 751-exact oracle: cos 0.999 — the pre-existing upstream padding-leak floor (SimpleDownsample folds one padded frame in; the shipped FmDecoder bucket pays the same). M5 Pro @s=1024: ANE 54.6 ms/step (was 286), 2591/2591 ops on ANE, zero fallback; Swift phys_footprint 25.5 MB steady (was 658 MB); core RTFx 13.5x oracle / 26.6x full bucket. The original graph on ANE is not just slow, it is broken: mel cos 0.68 and garbled whisper transcript vs 0.975 and an oracle-matching transcript for the rewrite. ANE-vs-CPU residual gap (log-mel 0.964 vs 0.999, -0.54 dB) is fp16 accumulation over 16 layers x 4 steps; the same package at CPU_ONLY matches the shipped conversion. GPU step is 23.5 ms vs 14.2 ms original (R=32 broadcast reduce): keep the original for the macOS GPU path, ship AneFmDecoder where ANE matters.
…t 72.7 MB / 244 MB steady, -0.90 dB; do not stack on ANE graph
int4 per-tensor -2.1dB, grouped needs iOS18 and still 2x 6-bit error, no benefit on the ANE path. Artifacts and script removed; measurement record kept in trials.md.
…rch -> 2.3ms GPU, 43dB parity, fully in-graph Whole decode as one fixed-shape mlprogram (33MB, iOS17 fp16): backbone, upsampler, both ISTFT heads with the ISTFT itself, 24k->48k resample, and the 12kHz crossover. - ISTFT (center): constant DFT matmul (hann folded) + 4-chunk shifted overlap-add + precomputed 1/envelope; 127.4dB vs torch.istft. logmag clamped pre-exp for fp16. - Resample: torchaudio sinc kernel (2x15 taps) as conv1d, bit-exact. - Crossover in-graph: linkwitz.py is an FFT brickwall (not IIR); rewritten as low + FIR_hp(high - low), 511-tap spectral-inversion highpass, 62dB vs the torch crossover on the oracle signal. Parity on the regenerated oracle mel: GPU 43.1dB wav SNR, log-mel cos 0.99982, RMS +0.000dB, whisper-base transcript identical. Latency: 2.34ms GPU / 10.56ms ANE @s=282 (torch cpu 77-81ms); S=555 variant 2.93ms. Fallback: 160/228 ops ANE, OLA pad/slice + snake tail on CPU; one ANE segment fails ANECCompile and falls back at runtime — GPU is the target. E2e: core 62ms + 2.3ms vocoder = ~64ms (was 139ms), oracle RTFx 21.5x -> 47x.
…ts port Dumps tokenizer ids, VocosFbank prompt mel, duration-expansion indices, anchor-Euler timesteps + a mini solver trajectory, and an end-to-end CoreML (gpu graphs + Vocoder555) stat block into the FluidAudio test resources. speed=1.0 (not upstream's hidden 1.3), target_rms=0.1.
…uxTTS phase 2)
Corpus, oracle and scoring pipeline for the FluidAudio Swift LuxTtsG2p:
- build_corpus.py: 1,000-sentence English gate corpus (100 MiniMax
conversational + 550 LibriSpeech test-clean + 250 numeric/currency/
time/date templates + 100 proper-name templates), plus a 499-sentence
dev split; both committed for reproducibility.
- validate.py: dump-oracle (EmiliaTokenizer espeak ground truth),
word-gap (whole-sentence vs word-joined measurement: 3.8% exact /
5.72% token edit - quantifies espeak's clause-level effects), and
score (gates: >=90% sentence exact, <=2% token edit).
- probe_lexicon.py: harvests 139k words x 7 espeak carrier contexts
(mid/final/unstr/vowel/pause/start/r) + capital rows + phrase-merge
table + homograph variants + en_list flag sets. Regenerates
lexicon_en_us.tsv (not committed; 3.7 MB, deterministic) and
lexicon_aux.json.
- reference_g2p.py: Python reference of the Swift runtime (unit
segmentation, expect_verb/noun/past counters per translateword.c,
position-aware $pause, right-to-left $strend2 resolution).
Scores 99.6% exact / 0.01% token edit on the gate corpus,
99.2% / 0.02% on dev.
- misaki_mapping.py: approach-B baseline (Misaki + symbol mapping),
measured corpus-wide at 0.5% exact / 9.75% token edit and rejected:
the divergence from espeak is lexical, not symbolic.
Oracle dumps are committed so the Swift gate is reproducible without
piper_phonemize:
fluidaudiocli luxtts-g2p-dump --corpus coreml/g2p/corpus_en_1000.txt \
--tokens tokens.txt --out swift.jsonl
python -m coreml.g2p.validate score --oracle coreml/g2p/oracle_tokens.jsonl \
--swift swift.jsonl
Alex-Wengg
added a commit
to FluidInference/FluidAudio
that referenced
this pull request
Jul 10, 2026
…785) ## Summary New TTS backend: **LuxTTS** (ZipVoice-Distill, Apache-2.0) — 48 kHz zero-shot voice cloning from a ~5 s reference clip, 4-step flow matching, fully CoreML. Requested in #49. Models: [FluidInference/luxtts-coreml](https://huggingface.co/FluidInference/luxtts-coreml) (converted in mobius PR FluidInference/mobius#75, full trial log there). ## What's included - **`LuxTtsManager`** (`Sources/FluidAudio/TTS/LuxTts/`): tokenizer → text encoder → host-side duration expansion + 4-step anchor-Euler solver → flow decoder → 48 kHz dual-head Vocos vocoder (all CoreML), volume matching, per-platform graph selection scaffold (gpu/ macOS today; ane/ wiring is a follow-up). - **Espeak-parity English G2P** (`TTS/LuxTts/G2p/`): 139k-word context-probed lexicon (0.94 MB bundled) + espeak clause-semantics engine. **99.6% sentence-exact / 0.01% token edit** vs the piper_phonemize oracle on a 1,000-sentence corpus; Swift byte-identical to its python reference. Plain text in — no espeak dependency. - **Native mel frontend** (`LuxTtsMelExtractor`, vDSP): VocosFbank parity max diff 1.7e-4. - **CLI**: `fluidaudiocli tts "text" --backend luxtts --prompt-audio ref.wav [--prompt-text] [--speed] [--seed] [--phonemes]`; prompt transcript auto-generated via Parakeet when `--prompt-text` omitted (lazy — no ASR download otherwise). Plus `luxtts-g2p-dump` for the reproducible G2P gate. - **Tests**: tokenizer/solver/expansion exact vs python fixtures; mel parity; G2P fixtures; e2e suite gated on model availability (`FLUIDAUDIO_RUN_LUXTTS_E2E=1`). - **Docs**: `Documentation/TTS/LuxTts.md`. ## Validation - E2E vs the python CoreML pipeline: output duration **exact** (220,672 samples), RMS +0.75 dB, Parakeet round-trip transcript **verbatim** on all fixtures. - Solver/expansion/timesteps: exact parity. G2P gate reproducible via committed corpus + scorer (mobius `models/tts/zipvoice/coreml/g2p/`). - Perf context (M5 Pro, from the mobius trials): decoder 14.7 ms/step GPU, vocoder 2.3 ms, e2e ~47× realtime; ANE-canonical graph (follow-up wiring) 27× RT at 25.5 MB footprint. ## Notes - Defaults chosen from root-caused upstream quirks: `speed` 1.0 (upstream's hidden ×1.3 clips sentence onsets) and guidance on trimming prompts at a speech pause (docs). - CoreML GPU stride-padded MLMultiArray outputs (100→112 floats/row) are handled with stride-aware copies — silent-garbage hazard worth knowing about for other backends. ## Follow-ups - iOS `ane/` graph host wiring (input layout differs) - >555-frame output chunking for long text - en-us-only G2P; other languages later
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CoreML conversion of LuxTTS (48 kHz zero-shot voice cloning; distilled ZipVoice, Apache-2.0, 4-step flow matching), requested in FluidInference/FluidAudio#49. Includes a full ANE-canonical rewrite of the Zipformer fm_decoder that takes ANE from broken to the best deployment path.
Model & destination
models/tts/zipvoice/— self-contained target per repo conventions (pyproject, conversion scripts, parity/bench harnesses, trials.md; upstream clones and build artifacts gitignored)--max-tokens/--max-frames); duration expansion, 4-step anchor-Euler solver, and vocoder stay host-sideValidation evidence (full history in
coreml/trials.md)Parity vs PyTorch: text encoder cos 0.999997; per-step decoder cos ≥0.9987; end-to-end log-mel cos 0.99925, RMS Δ −0.09 dB; whisper transcripts word-for-word identical to the torch oracle, including 10–14 s multi-sentence samples.
Performance (M5 Pro, 1024-frame bucket, Swift CoreML harness):
ANE-canonical rewrite (
coreml/ane/): (1,C,1,S) conv2d layout throughout, constant-buffer rel→abs attention (no gather; rank-32 SVD sharing, 88 MB total), depthwise k=31 split 15+15+1 (ANE kernel limit), 2591/2591 ops on ANE, exact fp32 parity (≤3.8e-4). The original graph is silently wrong on ANE (mel cos 0.68) — the rewrite is a correctness fix as much as a 5.2× speedup.Quantization: 6-bit palettization ships (101 MB weights, −0.45 dB, transcript-clean, 419 MB footprint); int8 parked (best quality but MPSGraph GPU crash on macOS 26.5); int4 evaluated and scrapped (record in trials.md).
Root-caused upstream quirk: onset word elision = prompt hard-cut mid-phrase (continuation behavior) amplified by a hidden speed×1.3; fixed by VAD-boundary prompt trimming + speed 1.0 — both defaults for the future Swift integration.
New dependencies
None repo-wide; the target's own venv uses torch/coremltools/lhotse/piper-phonemize (see pyproject). Note for FluidAudio integration: upstream frontend is espeak-based (GPL) — use the in-house G2P instead.
Follow-ups (not in this PR)
LuxTtsbackend🤖 Generated with Claude Code