feat(tts): LuxTTS backend — 48 kHz zero-shot voice cloning (CoreML)#785
Merged
Conversation
…hase 1, macOS) Adds the LuxTTS backend: TextEncoder -> 4-step anchor-Euler FmDecoder -> fixed-shape Vocos vocoder (48 kHz), conditioned on a <=5 s prompt clip + espeak-IPA transcript. Models from FluidInference/luxtts-coreml via ModelHub (gpu/ graph, .cpuAndGPU on macOS — the graph corrupts audio on the ANE; ane/ variant selection is structural, host wiring is phase 2). Host-side math mirrors upstream ZipVoice exactly and is fixture-pinned against the Python reference (mobius dump_swift_fixtures.py): - EmiliaTokenizer table + per-scalar espeak-IPA token ids (exact) - VocosFbank mel (24 kHz / n_fft 1024 / hop 256 / 100 mels, torchaudio periodic Hann + reflect pad + HTK fbanks, lhotse frame count): max_abs 1.7e-4 vs fixture (< 1e-3 gate) - solver timesteps (t_shift 0.5), ratio duration estimate, avg-duration expansion indices, anchor-Euler trajectory (exact) CoreML prediction outputs are stride-padded along the row dimension (100 -> 112 floats on GPU); outputs are compacted via strides before any host math — reading them contiguously produced silence-converged mel. E2E vs the Python CoreML pipeline (same prompt/phonemes/speed/seed accounting): 220,672 samples @48 kHz exact match, RMS +0.75 dB (< 1 dB gate), Parakeet ASR round-trip transcribes the test sentence verbatim. RTFx ~17x (debug build, M5 Pro). Phase 1 has no text->IPA frontend: synthesize(text:) throws g2pUnavailable. The Kokoro/StyleTTS2 Misaki frontend was evaluated and does not map onto tokens.txt (drops espeak length marks, uses ligatures + diphthong shorthands, different weak forms; 78 vs 85 tokens on the fixture sentence), so phase 1 ships the pre-phonemized path only: CLI --backend luxtts --prompt-audio --prompt-text with --phonemes. Speed default is 1.0 — upstream's hidden 1.3x squeeze clips sentence onsets. Prompts quieter than RMS 0.1 are boosted for conditioning and the output is scaled back (rms_norm contract); the CLI writes the WAV un-normalized to preserve prompt-matched loudness.
… (phase 2) LuxTTS was trained on espeak-ng (en-us) phonemes via EmiliaTokenizer, so raw-text input needs espeak parity, not a generic G2P. Approaches were measured on a 1,000-sentence espeak-oracle corpus (conversational + LibriSpeech + numbers/currency/dates + names): lexicon + clause rules (shipped): 99.6% sentence exact, 0.01% token edit naive word-by-word lexicon: 3.8% sentence exact, 5.72% token edit Misaki + symbol mapping (rejected): 0.5% sentence exact, 9.75% token edit Gates were >=90% / <=2%. The Misaki route fails because the divergence from espeak is lexical (vowel choices, stress positions, length marks), not symbolic — no mapping layer can close it. LuxTtsG2p (mirrors coreml/g2p/reference_g2p.py in mobius zipvoice): - 139k-word lexicon, each word probed offline in 7 espeak carrier contexts (mid / clause-final / before-only-unstressed / before-vowel / before-pause-word / clause-initial / before-r); sparse TSV bundled as 0.94 MB raw DEFLATE + 36 KB aux JSON, loaded lazily, no downloads. - espeak clause machinery ported from translate.c/dictionary.c semantics: multi-word merge entries (in the -> ɪnðə), right-to-left $strend2 stress resolution, expect_verb/noun/past homograph counters (to record vs the record), position-aware $pause (liaison/flap blocking), case-sensitive rows (I pronoun vs i letter), all-caps spell-out, camelCase splitting, punctuation semantics incl. espeak's glued-token quirks (date.It's -> 'dot'). - LuxTtsEnglishNormalizer: faithful port of upstream ZipVoice EnglishTextNormalizer (abbreviations + inflect-parity numbers, hyphens preserved because espeak merges twenty-six -> twˈɛntisˈɪks). The shared #711 normalizer is intentionally not reused: its different spellings would break espeak-oracle token parity. Wiring: - LuxTtsManager.synthesize(text:promptAudio:promptText:) is real now; LuxTtsError.g2pUnavailable is gone. - CLI: luxtts text mode needs no --phonemes (flag remains as an espeak- IPA bypass); --prompt-text is optional — when omitted the prompt clip is transcribed with Parakeet ASR (lazy: TTS-only users never download ASR models). - fluidaudiocli luxtts-g2p-dump: corpus -> phonemes/ids JSONL for the reproducible oracle gate (validate.py score in mobius zipvoice). Verification: - Swift dump is byte-identical to the Python reference on all 1,000 gate sentences; gates PASS (99.6% / 0.01%; dev 99.2% / 0.02%). - e2e text mode: all 3 phase-1 fixture sentences synthesized via CLI and round-tripped through Parakeet transcribe; transcripts match the input text (FluidAudio -> 'Fluid Audio' via espeak camel-split, comma placement aside). The no---prompt-text ASR path round-trips exactly. - LuxTtsG2pTests pin the fixture sentences (phonemes + token ids), normalizer year/ordinal/currency cases, weak forms/merges, strend2, homographs, all-caps, possessive fallback.
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 46.8s diarization time • Test runtime: 3m 17s • 07/08/2026, 04:58 PM EST |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 99.6s processing • Test runtime: 1m 51s • 07/08/2026, 04:55 PM EST |
Supertonic3 Smoke Test ✅
Runtime: 0m33s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 20s • 2026-07-08T21:00:12.156Z |
PocketTTS Smoke Test ✅
Runtime: 2m12s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 2m8s • 07/08/2026, 04:49 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 14m15s • 07/08/2026, 05:06 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Guards, reuse, and perf cleanups for the LuxTTS backend. None change synthesis output — verified via the standalone parity harness (solver timesteps + expansion indices byte-identical, mel max_abs 1.72e-4 < 1e-3) and the CLI e2e (838 frames / 220672 samples, RMS ~0.154, unchanged). 1. Solver: `tokensIndex` now throws `.degenerateDuration` when featuresLength < tokensCount (avg < 1 would collapse every frame onto the pad slot and synthesize silence). Normal-path math unchanged; throw threaded through the synthesizer call site. 2. Manager: `synthesize(text:)` fails fast with `.notInitialized` before the G2P lexicon load + phonemization, matching the phonemes path. 3. CLI: prompt ASR transcription runs before `manager.initialize()` (independent, only needs the prompt URL) and is wrapped to log an actionable "pass --prompt-text to skip ASR" message on failure. 4. G2p: `RawToken.isWordish` force-unwrap → guard-let (empty → false). 5. Synthesizer: `LuxTtsMelExtractor` built once and stored (Hann window, filterbank, DFT setup are all call-independent). Synthesizer becomes an actor to hold the non-Sendable extractor, mirroring StyleTTS2Synthesizer; extractor `fftSetup` is now `let`. 6. Solver loop: drop the redundant per-step tail-rezero (makeArray's reset already zeros the tail and nothing writes past activeCount); preallocate the `v` scratch once and reuse it via a `copyRows(into:)` overload instead of allocating per step. 7. Mel: vectorize the per-frame `log(max(x, floor))` with vDSP_vthr + vvlogf into a reused buffer (max_abs 1.72e-4, unchanged within fp tolerance vs fixture). Adds a unit test for the degenerate-duration guard.
The ane/ TextEncoder and FmDecoder graphs share a byte-for-byte identical external I/O contract with the gpu/ graphs (same input/output names, shapes and fp32 dtype) — the ANE canonicalization (channel-axis pre-concat, (1,C,1,S) working layout) lives entirely inside the MIL and transposes back to (1,1024,100) before output. So the iOS path needs no host-side adapter: the existing LuxTtsSynthesizer packing drives both variants unchanged. - ModelNames.LuxTts.defaultVariant already selects ane/ on iOS; add a FLUIDAUDIO_LUXTTS_VARIANT override so the ane/ path is exercisable on a Mac's ANE (validation seam, no #if os(iOS) fork). ModelHub's requiredFiles pattern filter already scopes downloads to the selected variant prefix, so macOS never pulls ane/ and iOS never pulls gpu/. - Extract the encoder/decoder compute-unit policy into a pure, unit-tested static (ane -> .cpuAndNeuralEngine, else .cpuAndGPU; explicit override wins). Vocoder stays .cpuAndGPU everywhere (only ~72% ANE-placeable and CPU_AND_NE compilation is flaky). - Document the shared ane/gpu I/O contract and the quality/latency tradeoff (~27x RT, ~25 MB ANE weights, ~-0.5 dB softer, transcript verbatim). Validated on this Mac's ANE (FmDecoder 100% ANE-resident, TextEncoder 99% via compute-plan): 48 kHz, 220672 samples (4.597s), RMS 0.157, Parakeet round-trip transcript verbatim. macOS gpu path byte-identical (no regress).
…xt ext)
Failure 1 (iOS build, region-based isolation error at the flow-matching
withUnsafeMutableBufferPointer/predict loop): drop the actor on
LuxTtsSynthesizer and make it a Sendable struct. The actor existed only to
store a non-Sendable LuxTtsMelExtractor; instead the extractor is now a
per-call local (so the struct is Sendable and safe to send across
LuxTtsManager's actor boundary), and its call-independent Hann-window /
mel-filterbank tables are hoisted into a process-wide Sendable cache so the
per-call construction only pays for the cheap vDSP DFT setup. Output is
byte-for-byte identical (verified 161792 samples, RMS 0.15827).
Failure 2 (LuxTtsG2pTests.testAllCapsSpellOut): the hand-authored expected
value ("ðə ...") was never run locally. The espeak oracle
(piper_phonemize en-us via EmiliaTokenizer) produces "ðɪ ˌɛfbˌiːˈaɪ kˈɔːld"
— the F=ɛf letter is vowel-initial, triggering the before-vowel weak form
of "the". Swift G2P already matches the oracle; fix the test expectation.
Re-verified every other string expectation in the file against the oracle;
all correct.
Failure 3 (ModelNamesTests.testModelFileExtensions): luxtts is the first
backend with a required .txt file (tokens.txt, a legitimate vocab file).
Add ".txt" to the test's validExtensions set (widen-only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New TTS backend: LuxTTS (ZipVoice-Distill, Apache-2.0) — 48 kHz zero-shot voice cloning from a ~5 s reference clip, 4-step flow matching, fully CoreML. Requested in #49.
Models: FluidInference/luxtts-coreml (converted in mobius PR FluidInference/mobius#75, full trial log there).
What's included
LuxTtsManager(Sources/FluidAudio/TTS/LuxTts/): tokenizer → text encoder → host-side duration expansion + 4-step anchor-Euler solver → flow decoder → 48 kHz dual-head Vocos vocoder (all CoreML), volume matching, per-platform graph selection scaffold (gpu/ macOS today; ane/ wiring is a follow-up).TTS/LuxTts/G2p/): 139k-word context-probed lexicon (0.94 MB bundled) + espeak clause-semantics engine. 99.6% sentence-exact / 0.01% token edit vs the piper_phonemize oracle on a 1,000-sentence corpus; Swift byte-identical to its python reference. Plain text in — no espeak dependency.LuxTtsMelExtractor, vDSP): VocosFbank parity max diff 1.7e-4.fluidaudiocli tts "text" --backend luxtts --prompt-audio ref.wav [--prompt-text] [--speed] [--seed] [--phonemes]; prompt transcript auto-generated via Parakeet when--prompt-textomitted (lazy — no ASR download otherwise). Plusluxtts-g2p-dumpfor the reproducible G2P gate.FLUIDAUDIO_RUN_LUXTTS_E2E=1).Documentation/TTS/LuxTts.md.Validation
models/tts/zipvoice/coreml/g2p/).Notes
speed1.0 (upstream's hidden ×1.3 clips sentence onsets) and guidance on trimming prompts at a speech pause (docs).Follow-ups
ane/graph host wiring (input layout differs)🤖 Generated with Claude Code