diff --git a/.gitignore b/.gitignore index 4b9c67ab..b0d3ac93 100644 --- a/.gitignore +++ b/.gitignore @@ -102,6 +102,14 @@ FluidAudioDatasets/ Resources/ !Sources/FluidAudio/Resources/ !Sources/FluidAudio/Resources/** +# LuxTts parity fixtures (dump_swift_fixtures.py); the reference wav +# stays untracked per the repo-wide *.wav rule (tests read JSON stats). +!Tests/FluidAudioTests/TTS/LuxTts/Resources/ +!Tests/FluidAudioTests/TTS/LuxTts/Resources/** +Tests/FluidAudioTests/TTS/LuxTts/Resources/*.wav +# LuxTts bundled G2P lexicon (probe_lexicon.py in mobius zipvoice) +!Sources/FluidAudio/TTS/LuxTts/G2p/Resources/ +!Sources/FluidAudio/TTS/LuxTts/G2p/Resources/** scripts/ !Scripts/parakeet_subset_benchmark.sh !Scripts/diarizer_subset_benchmark.sh diff --git a/Documentation/TTS/LuxTts.md b/Documentation/TTS/LuxTts.md new file mode 100644 index 00000000..6832868f --- /dev/null +++ b/Documentation/TTS/LuxTts.md @@ -0,0 +1,211 @@ +# LuxTTS (ZipVoice-Distill) Swift Inference + +Zero-shot voice-cloning TTS conditioned on a short prompt clip and its +transcript. 48 kHz mono Float32 output, 3-stage CoreML pipeline +(flow-matching, 4 anchor-Euler steps). + +## Overview + +LuxTTS is the CoreML port of ZipVoice-Distill (conversion lives in +`mobius/models/tts/zipvoice`). Each utterance runs: + +``` +espeak-IPA phonemes → TextEncoder → token_embeds [1, 256, 100] + avg-duration expansion → per-frame text_condition +prompt clip (24 kHz) → VocosFbank mel ×0.1 → speech_condition + FmDecoder ×4 anchor-Euler steps (guidance 3.0) + Vocos vocoder (fixed 282/555-frame bucket) → 48 kHz wav +``` + +Models: [FluidInference/luxtts-coreml](https://huggingface.co/FluidInference/luxtts-coreml) + +| Asset | Purpose | +|---|---| +| `gpu/TextEncoder.mlmodelc`, `gpu/FmDecoder.mlmodelc` | Original graph. **macOS** path, `.cpuAndGPU`. **Never run on the ANE** — the rel-pos attention path corrupts audio there. | +| `ane/TextEncoder.mlmodelc`, `ane/FmDecoder.mlmodelc` | ANE-canonical rewrite. **iOS** path, `.cpuAndNeuralEngine` (FmDecoder 100% ANE-resident, TextEncoder 99%). Same external I/O as `gpu/` — see below. | +| `vocoder/Vocoder282.mlmodelc`, `vocoder/Vocoder555.mlmodelc` | Fixed-shape 48 kHz Vocos vocoders (`mel (1,100,S) → audio`). Shared across variants; run `.cpuAndGPU` **everywhere** (only ~72% ANE-placeable, and CPU_AND_NE compilation is flaky). Smallest bucket ≥ generated frames is used; mel is never truncated. | +| `tokens.txt`, `config.json` | EmiliaTokenizer phoneme→id table (espeak IPA per Unicode scalar + pinyin initials/finals). Shared across variants. | + +### Platform variant selection + +`ModelNames.LuxTts.defaultVariant` picks `gpu` on macOS and `ane` on iOS; +`LuxTtsModelStore` maps that to `.cpuAndGPU` / `.cpuAndNeuralEngine`. The +downloader pulls **only** the selected variant's two `.mlmodelc` bundles +(plus the shared `vocoder/`, `tokens.txt`, `config.json`) — macOS never +fetches `ane/`, iOS never fetches `gpu/` (ModelHub's `requiredFiles` +pattern filter scopes the tree walk to the variant prefix). + +Set `FLUIDAUDIO_LUXTTS_VARIANT=ane` (or `gpu`) to override the platform +default. This is the validation seam: the M-series ANE runs the `ane/` +graph under `.cpuAndNeuralEngine` exactly as an iPhone would, so the iOS +path is fully exercisable on a Mac without an `#if os(iOS)` fork. + +### ANE vs GPU graph I/O (identical external contract) + +Despite the "ANE-canonical" rewrite (the FmDecoder pre-concatenates +`x|text_condition|speech_condition` on the channel axis and works in +`(1, C, 1, S)` form internally for zero-fallback ANE placement), **the +externally-visible input/output signature is byte-for-byte the same as the +`gpu/` graph** — the cat / transpose / expand-dims are all inside the MIL, +and the output is transposed back to `(1, 1024, 100)` before it leaves the +graph. So no host-side adapter is needed; the same `LuxTtsSynthesizer` +tensor packing drives both graphs. + +| model | inputs | output | +|---|---|---| +| **TextEncoder** (gpu + ane) | `tokens (1,256) int32`, `padding_mask (1,256) fp32` | `token_embeds (1,256,100) fp32` | +| **FmDecoder** (gpu + ane) | `x (1,1024,100)`, `text_condition (1,1024,100)`, `speech_condition (1,1024,100)`, `t (1)`, `guidance_scale (1)`, `padding_mask (1,1024)` — all fp32 | `v (1,1024,100) fp32` | + +### ANE quality / latency tradeoff + +The ANE FmDecoder runs in fp16 (ANE has no fp32 datapath); the gpu graph +keeps fp32 accumulation. This softens the output slightly but stays +transcript-verbatim: + +| variant | core RTFx (full bucket) | steady footprint | log-mel cos vs oracle | RMS delta | round-trip transcript | +|---|---|---|---|---|---| +| gpu (`.cpuAndGPU`) | ~92× | ~1 GB | 0.999 | -0.09 dB | verbatim | +| ane (`.cpuAndNeuralEngine`) | ~27× | ~660 MB / **25.5 MB** ANE weights | 0.964 | -0.54 dB | verbatim | + +The ANE path is ~0.5 dB softer, not bit-identical, but the Parakeet +round-trip transcript matches the input text exactly. The much smaller +jetsam-visible footprint is why iOS uses it. + +All shapes are fixed: ≤ 255 tokens (+1 pad slot), ≤ 1024 mel frames +total, ≤ 555 generated frames (~5.9 s per call; chunking is phase 2). + +## Quick Start + +### CLI + +```bash +swift run fluidaudiocli tts \ + "The quick brown fox jumps over the lazy dog, and honestly, it felt great." \ + --backend luxtts \ + --prompt-audio prompt_clip.wav \ + --prompt-text "quick brown fox jumps over the lazy dog and honestly it felt great." \ + --seed 42 \ + --output out.wav +``` + +`--prompt-audio` (voice to clone; first 5 s used) is required. Text and +`--prompt-text` are plain English — phonemized in-process by the +espeak-parity G2P (see below). If `--prompt-text` is omitted, the prompt +clip is transcribed with the built-in Parakeet ASR (models download on +first use; TTS-only runs never pay that cost). `--phonemes` bypasses the +G2P: both the text and `--prompt-text` are then espeak IPA (`en-us`). + +### Swift API + +```swift +let manager = try await LuxTtsManager.downloadAndCreate() +let result = try await manager.synthesize( + text: "The quick brown fox jumps over the lazy dog.", + promptAudio: promptURL, // any format/rate; 24 kHz mono internally + promptText: "The transcript of the prompt clip.", + speed: 1.0, + seed: 42) +// result.samples: 48 kHz mono Float32, prompt-matched loudness +``` + +Phoneme overloads remain for callers running their own espeak frontend: +`synthesize(phonemes:promptAudio:promptPhonemes:…)` and +`synthesize(tokenIds:promptAudio:promptTokenIds:…)` (raw `tokens.txt` ids). + +## Usage notes + +- **Keep `speed` at 1.0.** Upstream's `generate()` silently multiplies + speed by 1.3, which squeezes the ratio-based duration estimate and + clips sentence onsets. The Swift port never applies it. +- **Trim prompt silence.** Generated length is estimated as + `prompt_frames / prompt_tokens × text_tokens / speed`; leading or + trailing silence in the prompt inflates frames-per-token and slows or + pads the output. Trim with `VadManager` (or any editor) before + passing the clip. +- **Loudness contract**: prompts quieter than RMS 0.1 are boosted for + conditioning and the generated wav is scaled back to the prompt's + original level (upstream `rms_norm`). Don't peak-normalize the output. +- **Determinism**: same phonemes + prompt + seed → same output on the + same OS/hardware. The noise RNG is not torch's, so waveforms differ + from the Python pipeline while duration/loudness match (fixture-gated: + frame accounting exact, RMS within 1 dB). + +## G2P (phase 2): espeak-parity English frontend + +The model was trained on espeak-ng (`en-us`) phonemes via +EmiliaTokenizer, so `LuxTtsG2p` reproduces **espeak**, not a generic +G2P. It is a lexicon + rules engine, fully offline: + +- **Lexicon**: 139k words × up to 7 espeak-probed context variants + (mid-clause / clause-final / before-only-unstressed / before-vowel / + before-pause-word / clause-initial / before-r), harvested offline from + espeak-ng via piper_phonemize. Bundled as a 0.94 MB raw-DEFLATE + resource (3.7 MB expanded) + 36 KB aux tables — no downloads. +- **Clause rules** (ported from espeak's `translate.c`/`dictionary.c` + semantics): multi-word merge entries (`in the` → `ɪnðə`, `did not` → + `dɪdnˌɑːt`), `$strend2` stress resolution (right-to-left), homograph + verb/noun/past selection via `expect_verb/noun/past` counters + (`to record` → `ɹᵻkˈoːɹd` vs `the record` → `ɹˈɛkɚd`), position-aware + `$pause` handling (blocks liaison/flapping), linking-r, `the/to/a/an` + vowel-context forms, capital-sensitive rows (`I` pronoun vs `i` + letter, Polish/polish), all-caps spell-out (`FBI` → `ˌɛfbˌiːˈaɪ`), + camelCase splitting (`FluidAudio` → `flˈuːɪd ˈɔːdɪˌoʊ`). +- **Normalization**: faithful port of the upstream ZipVoice + `EnglishTextNormalizer` (abbreviations + inflect-parity numbers: + `$12.50` → `twelve dollars, fifty cents`, `1855` → `eighteen + fifty-five`, `21st` → `twenty-first`) — hyphens preserved because + espeak merges them without a space. + +**Measured against the espeak oracle** (1,000-sentence corpus: +conversational + LibriSpeech + numbers/dates/currency + names; +regenerate + score via `mobius/models/tts/zipvoice/coreml/g2p/`): + +| Approach | Sentence exact match | Token edit rate | +|---|---|---| +| **Lexicon + rules (shipped)** | **99.6%** (gate ≥ 90%) | **0.01%** (gate ≤ 2%) | +| naive word-by-word lexicon | 3.8% | 5.72% | +| Misaki + symbol mapping (rejected) | 0.5% | 9.75% | + +The Misaki mapping layer was measured corpus-wide and rejected: the +divergence from espeak is lexical (different vowel choices, stress +positions, missing length marks), not just symbolic, so no mapping can +close it. The gate is reproducible: + +```bash +swift run fluidaudiocli luxtts-g2p-dump --corpus corpus_en_1000.txt \ + --tokens tokens.txt --out swift_dump.jsonl +# in mobius/models/tts/zipvoice: +python -m coreml.g2p.validate score --oracle coreml/g2p/oracle_tokens.jsonl \ + --swift swift_dump.jsonl +``` + +OOV words fall back to possessive/plural suffix rules, camelCase/all-caps +handling, then letter spell-out; OOV token-id scalars are skipped with a +warning, matching upstream. + +## Tests + +```bash +swift test --filter LuxTts # tokenizer/solver/mel/G2P (fixtures) +FLUIDAUDIO_RUN_LUXTTS_E2E=1 swift test --filter LuxTtsE2ETests # model-dependent e2e + +# Exercise the iOS ane/ graph on a Mac's ANE (downloads the ane/ variant): +FLUIDAUDIO_LUXTTS_VARIANT=ane FLUIDAUDIO_RUN_LUXTTS_E2E=1 \ + swift test --filter LuxTtsE2ETests +``` + +Fixtures are generated by +`mobius/models/tts/zipvoice/coreml/dump_swift_fixtures.py` and live in +`Tests/FluidAudioTests/TTS/LuxTts/Resources/`. G2P expectations in +`LuxTtsG2pTests` are espeak-oracle outputs from +`mobius/models/tts/zipvoice/coreml/g2p/validate.py dump-oracle`; the +corpus-level gate is scored with `luxtts-g2p-dump` + `validate.py score` +(see the G2P section above). + +## Remaining TODOs + +- Long-input chunking across multiple vocoder windows (> 555 generated + frames currently errors; mel truncation is not allowed). +- Optional VAD-based automatic prompt-silence trimming. +- Non-English text (the G2P is `en-us` only; Mandarin pinyin tokens + exist in `tokens.txt` but have no frontend). diff --git a/Package.swift b/Package.swift index 62f1b26d..db938255 100644 --- a/Package.swift +++ b/Package.swift @@ -26,7 +26,10 @@ let package = Package( "FastClusterWrapper", "MachTaskSelfWrapper", ], - path: "Sources/FluidAudio" + path: "Sources/FluidAudio", + resources: [ + .copy("TTS/LuxTts/G2p/Resources") + ] ), .target( name: "FastClusterWrapper", @@ -52,6 +55,9 @@ let package = Package( dependencies: [ "FluidAudio", "FluidAudioCLI", + ], + resources: [ + .copy("TTS/LuxTts/Resources") ] ), ], diff --git a/Sources/FluidAudio/ModelNames.swift b/Sources/FluidAudio/ModelNames.swift index 188b56fa..b8ee1211 100644 --- a/Sources/FluidAudio/ModelNames.swift +++ b/Sources/FluidAudio/ModelNames.swift @@ -65,6 +65,13 @@ public enum Repo: String, CaseIterable, Sendable { /// recipe. Ships four `.mlmodelc` bundles + `tts.json` + /// `unicode_indexer.json` at the repo root. case supertonic3 = "FluidInference/supertonic-3-coreml" + /// LuxTTS (ZipVoice-Distill) — 48 kHz zero-shot voice-cloning TTS. + /// Two decoder graphs per fixed shape bucket: `gpu/` (original graph, + /// macOS GPU path) and `ane/` (ANE-canonical rewrite, iOS path), plus + /// fixed-shape Vocos vocoders under `vocoder/` and the shared + /// `tokens.txt` / `config.json`. Conversion lives in mobius + /// (`models/tts/zipvoice`). + case luxtts = "FluidInference/luxtts-coreml" /// Repository slug (without owner) public var name: String { @@ -133,6 +140,8 @@ public enum Repo: String, CaseIterable, Sendable { return "StyleTTS-2-coreml/iteration_3/compiled" case .supertonic3: return "supertonic-3-coreml" + case .luxtts: + return "luxtts-coreml" } } @@ -1132,6 +1141,74 @@ public enum ModelNames { } } + /// LuxTTS (ZipVoice-Distill) model names. The HF repo publishes the same + /// text encoder + flow-matching decoder in two graph layouts: + /// - `gpu/` — original graph; fastest on Mac GPU (do NOT run on ANE: + /// the seq-first rel-pos attention path corrupts audio there) + /// - `ane/` — ANE-canonical rewrite; 100% ANE placement (iOS path) + /// plus fixed-shape Vocos vocoders (282 / 555 generated frames) and the + /// shared `tokens.txt` / `config.json`. All decoder I/O is identical + /// across the two graphs. + public enum LuxTts { + public static let gpuVariant = "gpu" + public static let aneVariant = "ane" + + /// Platform default graph variant: `gpu/` on macOS, `ane/` elsewhere. + /// + /// The `FLUIDAUDIO_LUXTTS_VARIANT` environment variable overrides this + /// (accepts `gpu` / `ane`). It exists so the iOS `ane/` path can be + /// exercised on macOS — the M-series ANE runs the `ane/` graph under + /// `.cpuAndNeuralEngine` exactly as an iPhone would, so the override + /// is the on-device validation seam without an `#if os(iOS)` fork. + public static var defaultVariant: String { + if let override = variantOverride { return override } + #if os(macOS) + return gpuVariant + #else + return aneVariant + #endif + } + + /// `gpu` / `ane` parsed from `FLUIDAUDIO_LUXTTS_VARIANT`, or `nil` when + /// unset/unrecognized (falls back to the platform default). + static var variantOverride: String? { + guard + let raw = ProcessInfo.processInfo.environment["FLUIDAUDIO_LUXTTS_VARIANT"]? + .trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + else { return nil } + switch raw { + case gpuVariant: return gpuVariant + case aneVariant: return aneVariant + default: return nil + } + } + + public static let tokensFile = "tokens.txt" + public static let configFile = "config.json" + public static let vocoder282File = "vocoder/Vocoder282.mlmodelc" + public static let vocoder555File = "vocoder/Vocoder555.mlmodelc" + + public static func textEncoderFile(variant: String) -> String { + "\(variant)/TextEncoder.mlmodelc" + } + + public static func fmDecoderFile(variant: String) -> String { + "\(variant)/FmDecoder.mlmodelc" + } + + public static func requiredFiles(variant: String?) -> Set { + let v = variant ?? defaultVariant + return [ + textEncoderFile(variant: v), + fmDecoderFile(variant: v), + vocoder282File, + vocoder555File, + tokensFile, + configFile, + ] + } + } + /// Multilingual G2P (CharsiuG2P ByT5) model names public enum MultilingualG2P { public static let encoder = "MultilingualG2PEncoder" @@ -1359,6 +1436,9 @@ public enum ModelNames { } case .supertonic3: return ModelNames.Supertonic3.requiredFiles(veVariant: variant) + case .luxtts: + // Variants: "gpu" (macOS) / "ane" (iOS); nil → platform default. + return ModelNames.LuxTts.requiredFiles(variant: variant) } } } diff --git a/Sources/FluidAudio/TTS/LuxTts/G2p/LuxTtsEnglishNormalizer.swift b/Sources/FluidAudio/TTS/LuxTts/G2p/LuxTtsEnglishNormalizer.swift new file mode 100644 index 00000000..38b1d922 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/G2p/LuxTtsEnglishNormalizer.swift @@ -0,0 +1,238 @@ +import Foundation + +/// Port of ZipVoice's English text normalization (EmiliaTokenizer path: +/// `map_punctuations` + `zipvoice.tokenizer.normalizer.EnglishTextNormalizer`). +/// +/// LuxTTS must match the *upstream* normalizer exactly — the espeak oracle +/// the model was trained against runs behind it — so this is a faithful +/// port (inflect-parity number spelling, hyphens preserved: espeak merges +/// "twenty-six" -> `twˈɛntisˈɪks`), not a reuse of the repo's conservative +/// `EnglishTextNormalizer` (issue #711), whose different output would shift +/// tokens (e.g. "twenty six" with a space). +enum LuxTtsEnglishNormalizer { + + /// `EmiliaTokenizer.preprocess_text` + `EnglishTextNormalizer.normalize`. + static func normalize(_ text: String) -> String { + var result = mapPunctuations(text) + result = expandAbbreviations(result) + result = normalizeNumbers(result) + return result + } + + // MARK: - map_punctuations + + static func mapPunctuations(_ text: String) -> String { + var t = text + let pairs: [(String, String)] = [ + ("，", ","), ("。", "."), ("！", "!"), ("？", "?"), ("；", ";"), + ("：", ":"), ("、", ","), ("‘", "'"), ("“", "\""), ("”", "\""), + ("’", "'"), ("⋯", "…"), ("···", "…"), ("・・・", "…"), ("...", "…"), + ] + for (from, to) in pairs { + t = t.replacingOccurrences(of: from, with: to) + } + return t + } + + // MARK: - Abbreviations + + private static let abbreviations: [(NSRegularExpression, String)] = [ + ("mrs", "misess"), ("mr", "mister"), ("dr", "doctor"), + ("st", "saint"), ("co", "company"), ("jr", "junior"), + ("maj", "major"), ("gen", "general"), ("drs", "doctors"), + ("rev", "reverend"), ("lt", "lieutenant"), ("hon", "honorable"), + ("sgt", "sergeant"), ("capt", "captain"), ("esq", "esquire"), + ("ltd", "limited"), ("col", "colonel"), ("ft", "fort"), + ("etc", "et cetera"), ("btw", "by the way"), + ].map { (regex("\\b\($0.0)\\b", caseInsensitive: true), $0.1) } + + private static func expandAbbreviations(_ text: String) -> String { + var t = text + for (re, replacement) in abbreviations { + t = re.stringByReplacingMatches( + in: t, range: NSRange(location: 0, length: (t as NSString).length), + withTemplate: replacement) + } + return t + } + + // MARK: - Numbers (upstream regex order) + + private static let commaNumberRe = regex(#"([0-9][0-9\,]+[0-9])"#) + private static let poundsRe = regex(#"£([0-9\,]*[0-9]+)"#) + private static let dollarsRe = regex(#"\$([0-9\.\,]*[0-9]+)"#) + private static let fractionRe = regex(#"([0-9]+)/([0-9]+)"#) + private static let decimalRe = regex(#"([0-9]+\.[0-9]+)"#) + private static let percentRe = regex(#"([0-9\.\,]*[0-9]+%)"#) + private static let ordinalRe = regex(#"[0-9]+(st|nd|rd|th)"#) + private static let numberRe = regex(#"[0-9]+"#) + + private static func normalizeNumbers(_ text: String) -> String { + var t = text + t = replace(commaNumberRe, in: t) { g in g[1].replacingOccurrences(of: ",", with: "") } + t = replace(poundsRe, in: t) { g in "\(g[1]) pounds" } + t = replace(dollarsRe, in: t) { g in expandDollars(g[1]) } + t = replace(fractionRe, in: t) { g in + expandFraction(numerator: Int(g[1]) ?? 0, denominator: Int(g[2]) ?? 1) + } + t = replace(decimalRe, in: t) { g in g[1].replacingOccurrences(of: ".", with: " point ") } + t = replace(percentRe, in: t) { g in g[1].replacingOccurrences(of: "%", with: " percent") } + t = replace(ordinalRe, in: t) { g in " \(ordinalWords(g[0])) " } + t = replace(numberRe, in: t) { g in expandNumber(Int(g[0]) ?? 0) } + return t + } + + private static func expandDollars(_ match: String) -> String { + let parts = match.split(separator: ".", omittingEmptySubsequences: false) + if parts.count > 2 { return " \(match) dollars " } + let dollars = parts.count > 0 ? Int(parts[0]) ?? 0 : 0 + let cents = parts.count > 1 ? Int(parts[1]) ?? 0 : 0 + // Digits remain here on purpose (upstream behavior); the trailing + // cardinal pass expands them. + if dollars != 0 && cents != 0 { + let dollarUnit = dollars == 1 ? "dollar" : "dollars" + let centUnit = cents == 1 ? "cent" : "cents" + return " \(dollars) \(dollarUnit), \(cents) \(centUnit) " + } + if dollars != 0 { + return " \(dollars) \(dollars == 1 ? "dollar" : "dollars") " + } + if cents != 0 { + return " \(cents) \(cents == 1 ? "cent" : "cents") " + } + return " zero dollars " + } + + private static func expandFraction(numerator: Int, denominator: Int) -> String { + if numerator == 1 && denominator == 2 { return " one half " } + if numerator == 1 && denominator == 4 { return " one quarter " } + if denominator == 2 { return " \(cardinalWords(numerator)) halves " } + if denominator == 4 { return " \(cardinalWords(numerator)) quarters " } + return " \(cardinalWords(numerator)) \(ordinalize(cardinalWords(denominator))) " + } + + /// Upstream `_expand_number`: years 1000..<3000 use pair reading. + private static func expandNumber(_ num: Int) -> String { + if num > 1000 && num < 3000 { + if num == 2000 { return " two thousand " } + if num > 2000 && num < 2010 { + return " two thousand \(cardinalWords(num % 100)) " + } + if num % 100 == 0 { + return " \(cardinalWords(num / 100)) hundred " + } + // inflect group=2, zero='oh', comma replaced by space + let high = num / 100 + let low = num % 100 + let lowWords = low < 10 ? "oh \(cardinalWords(low))" : cardinalWords(low) + return " \(cardinalWords(high)) \(lowWords) " + } + return " \(cardinalWords(num)) " + } + + // MARK: - inflect-parity spelling + + private static let ones = [ + "zero", "one", "two", "three", "four", "five", "six", "seven", + "eight", "nine", "ten", "eleven", "twelve", "thirteen", "fourteen", + "fifteen", "sixteen", "seventeen", "eighteen", "nineteen", + ] + private static let tens = [ + "", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", + "eighty", "ninety", + ] + private static let scales: [(Int, String)] = [ + (1_000_000_000_000, "trillion"), (1_000_000_000, "billion"), + (1_000_000, "million"), (1_000, "thousand"), + ] + + /// `inflect.number_to_words(n, andword="")`: hyphenated tens, scale + /// groups joined with ", " (only between non-zero groups). + static func cardinalWords(_ n: Int) -> String { + if n < 0 { return "minus \(cardinalWords(-n))" } + if n < 20 { return ones[n] } + if n < 100 { + let t = tens[n / 10] + return n % 10 == 0 ? t : "\(t)-\(ones[n % 10])" + } + if n < 1000 { + let head = "\(ones[n / 100]) hundred" + return n % 100 == 0 ? head : "\(head) \(cardinalWords(n % 100))" + } + var remaining = n + var groups: [String] = [] + for (value, name) in scales { + if remaining >= value { + groups.append("\(cardinalWords(remaining / value)) \(name)") + remaining %= value + } + } + if remaining > 0 { groups.append(cardinalWords(remaining)) } + return groups.joined(separator: ", ") + } + + /// `inflect.number_to_words("42nd")` — cardinal words with the final + /// word ordinalized ("forty-second"). + static func ordinalWords(_ ordinalDigits: String) -> String { + let digits = String(ordinalDigits.prefix(while: { $0.isNumber })) + guard let n = Int(digits) else { return ordinalDigits } + return ordinalize(cardinalWords(n)) + } + + /// Ordinalize the last word of a cardinal phrase (inflect.ordinal). + static func ordinalize(_ words: String) -> String { + let irregular: [String: String] = [ + "one": "first", "two": "second", "three": "third", + "five": "fifth", "eight": "eighth", "nine": "ninth", + "twelve": "twelfth", + ] + // Find the last word (after the last space or hyphen). + var separator: Character = " " + var lastRange = words.startIndex.. NSRegularExpression { + // Compile-time constant patterns; a failure is a programmer error. + try! NSRegularExpression( + pattern: pattern, options: caseInsensitive ? [.caseInsensitive] : []) + } + + private static func replace( + _ re: NSRegularExpression, in text: String, + transform: ([String]) -> String + ) -> String { + let ns = text as NSString + let matches = re.matches(in: text, range: NSRange(location: 0, length: ns.length)) + guard !matches.isEmpty else { return text } + let mutable = NSMutableString(string: text) + for match in matches.reversed() { + var groups: [String] = [] + groups.reserveCapacity(match.numberOfRanges) + for index in 0.. ɪnðə, "did not" -> dɪdnˌɑːt) +/// * homograph verb/noun/past selection driven by espeak's +/// expect_verb/noun/past counters ($verbf/$nounf/$pastf words) +/// * position-dependent $pause behavior (liaison/flapping blocking) +/// * punctuation semantics (",;:" emit + space, ".!?" emit + no space, +/// "…" silent break, quotes = pause boundary, mid-token "." = "dot") +/// +/// The algorithm mirrors `mobius/models/tts/zipvoice/coreml/g2p/ +/// reference_g2p.py`, which is score-gated against the espeak oracle +/// (1,000-sentence corpus: 99.6% sentence exact match, 0.01% token edit +/// rate — see Documentation/TTS/LuxTts.md). +public struct LuxTtsG2p: Sendable { + + // MARK: - Data + + /// Sparse variant row; `nil` means "same as mid". + struct Row: Sendable { + let mid: String + let final: String? + let unstr: String? + let vowel: String? + let pause: String? + let start: String? + let rvar: String? + } + + struct PhraseEntry: Sendable { + let mid: String? + let midcap: String? + let final: String? + let vowel: String? + let start: String? + let startcap: String? + let rvar: String? + let atEndOnly: Bool + let flagWords: [String] // constituent words, for counter updates + + var usableMidVariant: Bool { + mid != nil || midcap != nil || vowel != nil || rvar != nil + } + } + + let entries: [String: Row] + let phrases: [String: PhraseEntry] + let maxPhraseLength: Int + let homographs: [String: [String: String]] + let verbf: Set + let verbsf: Set + let nounf: Set + let pastf: Set + let verbextend: Set + let pauseWords: Set + let allcapsWords: Set + let letters: [String: String] + + private static let clausePunct = Set(",.!?;:") + private static let vowelScalars = Set("aeiouæɑɐɔəɛɜɪʊʌʉɒɚᵻ") + private static let stressMarks = Set("ˈˌ") + private static let voiceless = Set("ptkfθsʃ") + private static let sibilant = Set("szʃʒ") + + // MARK: - Loading + + /// Load the bundled lexicon + aux tables. + public init() throws { + guard + let lexiconURL = Bundle.module.url( + forResource: "luxtts_en_us_lexicon.tsv", withExtension: "zz", + subdirectory: "Resources"), + let auxURL = Bundle.module.url( + forResource: "luxtts_en_us_g2p_aux", withExtension: "json", + subdirectory: "Resources") + else { + throw LuxTtsError.tokenizerFailed("bundled G2P resources missing") + } + try self.init(lexiconURL: lexiconURL, auxURL: auxURL) + } + + init(lexiconURL: URL, auxURL: URL) throws { + let compressed = try Data(contentsOf: lexiconURL) + let tsvData: Data + do { + tsvData = try (compressed as NSData).decompressed(using: .zlib) as Data + } catch { + throw LuxTtsError.tokenizerFailed( + "cannot decompress lexicon: \(error.localizedDescription)") + } + guard let tsv = String(data: tsvData, encoding: .utf8) else { + throw LuxTtsError.tokenizerFailed("lexicon is not UTF-8") + } + + var entries: [String: Row] = [:] + entries.reserveCapacity(140_000) + for line in tsv.split(separator: "\n", omittingEmptySubsequences: true) { + let cols = line.split(separator: "\t", omittingEmptySubsequences: false) + .map(String.init) + guard cols.count >= 2 else { continue } + func col(_ i: Int) -> String? { + i < cols.count && !cols[i].isEmpty ? cols[i] : nil + } + entries[cols[0]] = Row( + mid: cols[1], final: col(2), unstr: col(3), vowel: col(4), + pause: col(5), start: col(6), rvar: col(7)) + } + self.entries = entries + + let auxData = try Data(contentsOf: auxURL) + guard + let aux = try JSONSerialization.jsonObject(with: auxData) as? [String: Any], + let phrasesJson = aux["phrases"] as? [String: [String: Any]], + let homographsJson = aux["homographs"] as? [String: [String: String]], + let flagSets = aux["flag_sets"] as? [String: [String]], + let letters = aux["letters"] as? [String: String] + else { + throw LuxTtsError.tokenizerFailed("malformed G2P aux JSON") + } + + var phrases: [String: PhraseEntry] = [:] + var maxLen = 1 + for (key, value) in phrasesJson { + let flags = value["flags"] as? [String] ?? [] + phrases[key] = PhraseEntry( + mid: value["mid"] as? String, + midcap: value["midcap"] as? String, + final: value["final"] as? String, + vowel: value["vowel"] as? String, + start: value["start"] as? String, + startcap: value["startcap"] as? String, + rvar: value["r"] as? String, + atEndOnly: flags.contains("$atend"), + flagWords: key.split(separator: " ").map(String.init)) + maxLen = max(maxLen, key.split(separator: " ").count) + } + self.phrases = phrases + self.maxPhraseLength = maxLen + self.homographs = homographsJson + self.verbf = Set(flagSets["verbf"] ?? []) + self.verbsf = Set(flagSets["verbsf"] ?? []) + self.nounf = Set(flagSets["nounf"] ?? []) + self.pastf = Set(flagSets["pastf"] ?? []) + self.verbextend = Set(flagSets["verbextend"] ?? []) + self.pauseWords = Set(flagSets["pause"] ?? []).union(Set(flagSets["brk"] ?? [])) + self.allcapsWords = Set(flagSets["allcaps"] ?? []) + .subtracting(Set(flagSets["abbrev"] ?? [])) + self.letters = letters + } + + // MARK: - Public API + + /// English text → espeak-IPA phoneme string (the `tokens.txt` scalar + /// set). Includes the upstream ZipVoice text normalization. + public func phonemize(text: String) -> String { + phonemizeNormalized(LuxTtsEnglishNormalizer.normalize(text)) + } + + // MARK: - Tokenization + + private enum Item { + case word(String) + case hyph([String]) + case literal(String) + } + + private struct RawToken { + let text: String // word, digit run, or single punctuation scalar + let start: Int // scalar offset + let end: Int + var isWordish: Bool { + guard let c = text.unicodeScalars.first else { return false } + return CharacterSet.letters.contains(c) || CharacterSet.decimalDigits.contains(c) + } + } + + private func rawTokens(_ text: String) -> [RawToken] { + var tokens: [RawToken] = [] + let scalars = Array(text.unicodeScalars) + var i = 0 + while i < scalars.count { + let c = scalars[i] + if CharacterSet.whitespacesAndNewlines.contains(c) { + i += 1 + continue + } + let start = i + if CharacterSet.letters.contains(c), c.isASCII { + i += 1 + while i < scalars.count, + (CharacterSet.letters.contains(scalars[i]) && scalars[i].isASCII) + || scalars[i] == "'" + { + i += 1 + } + } else if CharacterSet.decimalDigits.contains(c) { + i += 1 + while i < scalars.count, CharacterSet.decimalDigits.contains(scalars[i]) { + i += 1 + } + } else { + i += 1 + } + tokens.append( + RawToken( + text: String(String.UnicodeScalarView(scalars[start.. String { + let raw = rawTokens(normalized) + + // hyphen chains: word(-word)+ with no spaces + var tokens: [(item: Item?, punct: String?, start: Int, end: Int)] = [] + var i = 0 + while i < raw.count { + let tok = raw[i] + if tok.isWordish, isAlphaStart(tok.text), + i + 2 < raw.count, raw[i + 1].text == "-", + raw[i + 1].start == tok.end, raw[i + 2].start == raw[i + 1].end, + isAlphaStart(raw[i + 2].text) + { + var parts = [tok.text] + var j = i + 1 + var lastEnd = tok.end + while j + 1 < raw.count, raw[j].text == "-", raw[j].start == lastEnd, + isAlphaStart(raw[j + 1].text), raw[j + 1].start == raw[j].end + { + parts.append(raw[j + 1].text) + lastEnd = raw[j + 1].end + j += 2 + } + tokens.append((.hyph(parts), nil, tok.start, lastEnd)) + i = j + continue + } + if tok.isWordish { + tokens.append((.word(tok.text), nil, tok.start, tok.end)) + } else { + tokens.append((nil, tok.text, tok.start, tok.end)) + } + i += 1 + } + + var pieces: [String] = [] + var clause: [Item] = [] + var breaks: [Bool] = [] + var clauseInitial = true + var pendingBreak = false + + func flush() { + defer { + clause.removeAll() + breaks.removeAll() + pendingBreak = false + } + guard !clause.isEmpty else { return } + let chunks = phonemizeClause( + items: clause, clauseInitial: clauseInitial, breakBefore: breaks) + let phon = chunks.filter { !$0.isEmpty }.joined(separator: " ") + if !phon.isEmpty { pieces.append(phon) } + } + + for (idx, entry) in tokens.enumerated() { + if let item = entry.item { + clause.append(item) + breaks.append(pendingBreak) + pendingBreak = false + continue + } + guard var ch = entry.punct else { continue } + if ch == "—" || ch == "–" { ch = ";" } + + let prevGlued = idx > 0 && tokens[idx - 1].end == entry.start + let nextGlued = idx + 1 < tokens.count && tokens[idx + 1].start == entry.end + let midToken = prevGlued && nextGlued + + if Self.clausePunct.contains(Character(ch)) || ch == "…" { + if midToken { + // espeak reads glued "." as "dot", "!" as "exclamation"; + // "," is dropped; "?" becomes a silent join + switch ch { + case ".": + clause.append(.literal("dˈɑːt")) + breaks.append(false) + continue + case "!": + clause.append(.literal("ˈɛkskləmˌeɪʃən")) + breaks.append(false) + continue + case ",": + continue + case "?": + flush() + clauseInitial = true + continue + default: + break + } + } + flush() + if ch != "…" { + pieces.append(ch) + if ",;:".contains(ch) { pieces.append(" ") } + } + clauseInitial = true + } else if "\"'()[]«»".contains(ch) { + // quotes/parens: transparent, but a pause boundary + if !clause.isEmpty { pendingBreak = true } + } + // dashes/other symbols: transparent + } + flush() + + var text = pieces.joined() + while text.contains(" ") { + text = text.replacingOccurrences(of: " ", with: " ") + } + return text.trimmingCharacters(in: .whitespaces) + } + + private func isAlphaStart(_ s: String) -> Bool { + guard let c = s.unicodeScalars.first else { return false } + return CharacterSet.letters.contains(c) + } + + // MARK: - Clause translation + + private enum UnitKind { + case word + case hyph + case literal + case phrase(PhraseEntry) + } + + private func phonemizeClause( + items: [Item], clauseInitial: Bool, breakBefore: [Bool] + ) -> [String] { + let n = items.count + guard n > 0 else { return [] } + + // unit segmentation: greedy phrase match over consecutive words, + // consumed only when usable at its position + var units: [(kind: UnitKind, start: Int, length: Int)] = [] + var i = 0 + while i < n { + switch items[i] { + case .hyph: + units.append((.hyph, i, 1)) + i += 1 + continue + case .literal: + units.append((.literal, i, 1)) + i += 1 + continue + case .word: + break + } + var matched: (PhraseEntry, Int)? = nil + let maxLen = min(maxPhraseLength, n - i) + if maxLen >= 2 { + for length in stride(from: maxLen, through: 2, by: -1) { + var words: [String] = [] + var ok = true + for k in i..<(i + length) { + guard case .word(let w) = items[k] else { + ok = false + break + } + if k > i && breakBefore[k] { + ok = false + break + } + words.append(w.lowercased()) + } + guard ok, let entry = phrases[words.joined(separator: " ")] else { continue } + let atEnd = i + length == n + let usable = + entry.usableMidVariant + || (atEnd && entry.final != nil) + || (clauseInitial && i == 0 + && (entry.start != nil || entry.startcap != nil)) + let atEndSatisfied = !entry.atEndOnly || atEnd || entry.usableMidVariant + if usable && atEndSatisfied { + matched = (entry, length) + break + } + } + } + if let (entry, length) = matched { + units.append((.phrase(entry), i, length)) + i += length + } else { + units.append((.word, i, 1)) + i += 1 + } + } + + // pass 1 (left to right): homograph counters + $pause application + var flags = [String?](repeating: nil, count: units.count) + var pauseApplied = [Bool](repeating: false, count: units.count) + var counters = Counters() + var wordPos = 0 + for (u, unit) in units.enumerated() { + if breakBefore[unit.start] { wordPos = 0 } + if case .word = unit.kind, case .word(let token) = items[unit.start] { + flags[u] = counters.homographKey(token: token) + if pauseWords.contains(token.lowercased()), wordPos >= 2, + unit.start != n - 1 + { + pauseApplied[u] = true + wordPos = 0 // the pause word restarts the count as word 0 + } + } + for k in unit.start..<(unit.start + unit.length) { + if case .word(let w) = items[k] { + updateCounters(&counters, token: w) + } + if !pauseApplied[u] { wordPos += 1 } + } + } + + // pass 2 (right to left): variant selection ($strend2 chains + // resolve on the *selected* stress states of following words) + var chunks = [String](repeating: "", count: units.count) + var stressedAfter = false + var nextFirstPhone: Character? = nil + var nextIsBreak = false + + for u in stride(from: units.count - 1, through: 0, by: -1) { + let unit = units[u] + let atEnd = unit.start + unit.length == n + let atStart = clauseInitial && unit.start == 0 + let chunk: String + switch unit.kind { + case .phrase(let entry): + var tokens: [String] = [] + for k in unit.start..<(unit.start + unit.length) { + if case .word(let w) = items[k] { tokens.append(w) } + } + chunk = selectPhrase( + entry, tokens: tokens, atEnd: atEnd, atStart: atStart, + nextFirstPhone: nextFirstPhone, nextIsBreak: nextIsBreak) + case .hyph: + guard case .hyph(let parts) = items[unit.start] else { continue } + chunk = selectHyph( + parts, flag: flags[u], atEnd: atEnd, atStart: atStart, + nextFirstPhone: nextFirstPhone, nextIsBreak: nextIsBreak, + stressedAfter: stressedAfter) + case .literal: + guard case .literal(let pron) = items[unit.start] else { continue } + chunk = pron + case .word: + guard case .word(let token) = items[unit.start] else { continue } + chunk = selectWord( + token, flag: flags[u], atEnd: atEnd, atStart: atStart, + nextFirstPhone: nextFirstPhone, nextIsBreak: nextIsBreak, + stressedAfter: stressedAfter) + } + chunks[u] = chunk + if chunk.contains("ˈ") { stressedAfter = true } + nextFirstPhone = firstPhone(chunk) + nextIsBreak = pauseApplied[u] || breakBefore[unit.start] + } + return chunks + } + + // MARK: - espeak expect_verb/noun/past counters (translateword.c) + + private struct Counters { + var verb = 0 + var verbS = 0 + var noun = 0 + var past = 0 + + func homographKey(token: String) -> String? { + if verb > 0 || (verbS > 0 && token.lowercased().hasSuffix("s")) { + return "verb" + } + if past > 0 { return "past" } + if noun > 0 { return "noun" } + return nil + } + } + + private static let contractionBase: [String: String] = [ + "won't": "will", "can't": "can", "shan't": "shall", + ] + + /// The word whose espeak flags drive the counters; n't/'ll/'ve/'s + /// contractions inherit the auxiliary's flags (doesn't -> does). + private func flagWord(for token: String) -> String? { + let lower = token.lowercased().replacingOccurrences(of: "’", with: "'") + if lower == "i" && token != "I" { + return nil // lowercase i is the letter, not the pronoun + } + if let base = Self.contractionBase[lower] { return base } + if lower.hasSuffix("n't") { return String(lower.dropLast(3)) } + if lower.hasSuffix("'ll") { return "will" } + if lower.hasSuffix("'ve") { return "have" } + if lower.hasSuffix("'s") { return "is" } + return lower + } + + private func updateCounters(_ counters: inout Counters, token: String) { + let word = flagWord(for: token) + if let word { + if pastf.contains(word) { + counters.past = 3 + counters.verb = 0 + counters.noun = 0 + } else if verbf.contains(word) { + counters.verb = 2 + counters.verbS = 0 + counters.noun = 0 + } else if verbsf.contains(word) { + counters.verb = 0 + counters.verbS = 2 + counters.past = 0 + counters.noun = 0 + } else if nounf.contains(word) { + counters.noun = 2 + counters.verb = 0 + counters.verbS = 0 + counters.past = 0 + } + } + if word == nil || !verbextend.contains(word!) { + if counters.verb > 0 { counters.verb -= 1 } + if counters.verbS > 0 { counters.verbS -= 1 } + if counters.noun > 0 { counters.noun -= 1 } + if counters.past > 0 { counters.past -= 1 } + } + } + + // MARK: - Variant selection + + /// Case-sensitive row first (I vs i, Polish vs polish), then lower-case. + func lookup(_ word: String) -> Row? { + let normalized = word.replacingOccurrences(of: "’", with: "'") + return entries[normalized] ?? entries[normalized.lowercased()] + } + + private func firstPhone(_ pron: String) -> Character? { + pron.first { !Self.stressMarks.contains($0) } + } + + private func selectWord( + _ token: String, flag: String?, atEnd: Bool, atStart: Bool, + nextFirstPhone: Character?, nextIsBreak: Bool, stressedAfter: Bool + ) -> String { + // all-caps tokens spell out unless espeak marks them $allcaps words + if token.count >= 2, token == token.uppercased(), + token.allSatisfy({ $0.isLetter }), + !allcapsWords.contains(token.lowercased()) + { + return spellLetters(token) + } + + if let flag, let variants = homographs[token.lowercased()], + let form = variants[flag] + { + return form + } + + guard let row = lookup(token) else { return oovPron(token) } + + if atEnd { return row.final ?? row.mid } + + let nextIsVowel = nextFirstPhone.map { Self.vowelScalars.contains($0) } ?? false + + if let unstr = row.unstr, !stressedAfter, !nextIsBreak { + // liaison/flap still applies on top of the $strend2 form + var selected = unstr + if let vowel = row.vowel, nextIsVowel, let last = row.mid.last { + if vowel == row.mid + "ɹ", selected.hasSuffix(String(last)) { + selected += "ɹ" + } else if vowel == String(row.mid.dropLast()) + "ɾ", + selected.hasSuffix(String(last)) + { + selected = String(selected.dropLast()) + "ɾ" + } + } + return selected + } + if nextIsBreak { return row.pause ?? row.mid } + if nextFirstPhone == "ɹ", let rvar = row.rvar { return rvar } + if nextIsVowel, let vowel = row.vowel { return vowel } + if atStart, let start = row.start { return start } + return row.mid + } + + private func selectPhrase( + _ entry: PhraseEntry, tokens: [String], atEnd: Bool, atStart: Bool, + nextFirstPhone: Character?, nextIsBreak: Bool + ) -> String { + let capitalized = tokens.first?.first?.isUppercase ?? false + if atStart, capitalized, let startcap = entry.startcap { return startcap } + if atStart, let start = entry.start { return start } + if atEnd, let final = entry.final { return final } + let nextIsVowel = nextFirstPhone.map { Self.vowelScalars.contains($0) } ?? false + if !atEnd, !nextIsBreak { + if nextFirstPhone == "ɹ", let rvar = entry.rvar { return rvar } + if nextIsVowel, let vowel = entry.vowel { return vowel } + } + if capitalized, !atEnd, let midcap = entry.midcap { return midcap } + if !atEnd, let mid = entry.mid { return mid } + // fall back to word-by-word composition + var chunks: [String] = [] + for (index, word) in tokens.enumerated() { + guard let row = lookup(word) else { + chunks.append(oovPron(word)) + continue + } + if index == tokens.count - 1 && atEnd { + chunks.append(row.final ?? row.mid) + } else { + chunks.append(row.mid) + } + } + return chunks.joined(separator: " ") + } + + /// Hyphen chain: whole-token row if probed (twenty-six), else parts + /// pronounced separately and concatenated without a space. + private func selectHyph( + _ parts: [String], flag: String?, atEnd: Bool, atStart: Bool, + nextFirstPhone: Character?, nextIsBreak: Bool, stressedAfter: Bool + ) -> String { + let key = parts.joined(separator: "-") + if lookup(key) != nil { + return selectWord( + key, flag: flag, atEnd: atEnd, atStart: atStart, + nextFirstPhone: nextFirstPhone, nextIsBreak: nextIsBreak, + stressedAfter: stressedAfter) + } + var out: [String] = [] + for (index, part) in parts.enumerated() { + if index == parts.count - 1 { + out.append( + selectWord( + part, flag: nil, atEnd: atEnd, atStart: false, + nextFirstPhone: nextFirstPhone, nextIsBreak: nextIsBreak, + stressedAfter: stressedAfter)) + } else if let row = lookup(part) { + out.append(row.mid) + } else { + out.append(oovPron(part)) + } + } + return out.joined() + } + + // MARK: - Fallbacks + + private func spellLetters(_ word: String) -> String { + let letters = word.lowercased().filter { $0.isLetter } + var out = "" + for (index, letter) in letters.enumerated() { + let name = self.letters[String(letter)] ?? "" + let mark = index == letters.count - 1 ? "ˈ" : "ˌ" + out += letterNameWithStress(name, mark: mark) + } + return out + } + + private func letterNameWithStress(_ name: String, mark: String) -> String { + let stripped = name.filter { !Self.stressMarks.contains($0) } + if let index = stripped.firstIndex(where: { Self.vowelScalars.contains($0) }) { + return String(stripped[.. String { + guard let last = mid.last else { return mid } + if Self.sibilant.contains(last) { return mid + "ɪz" } + if Self.voiceless.contains(last) { return mid + "s" } + return mid + "z" + } + + func oovPron(_ word: String) -> String { + let lower = word.lowercased().replacingOccurrences(of: "’", with: "'") + for suffix in ["'s", "s'"] where lower.hasSuffix(suffix) { + if let base = lookup(String(lower.dropLast(suffix.count))) { + return suffixS(base.mid) + } + } + if lower.hasSuffix("s"), let base = lookup(String(lower.dropLast())) { + return suffixS(base.mid) + } + if word.count >= 2, word == word.uppercased(), !word.contains("'") { + return spellLetters(word) + } + // camelCase: split at case boundaries, parts space-joined, + // single letters spoken as letter names (iPhone -> ˈaɪ fˈoʊn) + let parts = camelParts(word) + if parts.count > 1 { + var out: [String] = [] + for part in parts { + if part.count == 1, part.first!.isLetter { + out.append( + letterNameWithStress( + letters[part.lowercased()] ?? "", mark: "ˈ")) + } else if let row = lookup(part) { + out.append(row.mid) + } else { + out.append(oovPron(part)) + } + } + return out.joined(separator: " ") + } + return spellLetters(word) + } + + /// `[A-Z]?[a-z']+|[A-Z]+(?![a-z])` — FluidAudio -> [Fluid, Audio]. + private func camelParts(_ word: String) -> [String] { + var parts: [String] = [] + var current = "" + let chars = Array(word) + var i = 0 + while i < chars.count { + let c = chars[i] + if c.isUppercase { + // uppercase run: attach the last capital to a following + // lowercase run (HTTPServer -> HTTP, Server) + var run = "" + while i < chars.count, chars[i].isUppercase { + run.append(chars[i]) + i += 1 + } + let nextIsLower = i < chars.count && (chars[i].isLowercase || chars[i] == "'") + if nextIsLower { + if run.count > 1 { parts.append(String(run.dropLast())) } + current = String(run.last!) + } else { + parts.append(run) + current = "" + } + } else { + current.append(c) + i += 1 + while i < chars.count, chars[i].isLowercase || chars[i] == "'" { + current.append(chars[i]) + i += 1 + } + parts.append(current) + current = "" + } + } + if !current.isEmpty { parts.append(current) } + return parts.filter { !$0.isEmpty } + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_g2p_aux.json b/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_g2p_aux.json new file mode 100644 index 00000000..fd925074 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_g2p_aux.json @@ -0,0 +1,2251 @@ +{ + "phrases": { + "bon voyage": { + "mid": "bˌɑːn vɔɪˈɑːʒ", + "final": "bˌɑːn vɔɪˈɑːʒ", + "vowel": "bˌɑːn vɔɪˈɑːʒ", + "flags": [] + }, + "bow down": { + "mid": "bˌaʊ dˈaʊn", + "final": "bˌaʊ dˈaʊn", + "vowel": "bˌaʊ dˈaʊn", + "flags": [] + }, + "coup de grace": { + "mid": "kˌuːdə ɡɹˈɑː", + "final": "kˌuːdə ɡɹˈɑː", + "vowel": "kˌuːdə ɡɹˈɑː", + "flags": [] + }, + "cum laude": { + "mid": "kʊmlˈaʊdi", + "final": "kʊmlˈaʊdi", + "vowel": "kʊmlˈaʊdi", + "flags": [] + }, + "de gallo": { + "mid": "dəɡˈaɪoʊ", + "final": "dəɡˈaɪoʊ", + "vowel": "dəɡˈaɪoʊ", + "flags": [] + }, + "de jure": { + "mid": "deɪdʒˈʊɹɹi", + "final": "deɪdʒˈʊɹɹi", + "vowel": "deɪdʒˈʊɹɹi", + "flags": [] + }, + "en masse": { + "mid": "ɑːn mˈæs", + "final": "ɑːn mˈæs", + "vowel": "ɑːn mˈæs", + "flags": [] + }, + "en route": { + "mid": "ɑːn ɹˈuːt", + "final": "ɑːn ɹˈuːt", + "vowel": "ɑːn ɹˈuːt", + "flags": [] + }, + "faux pas": { + "mid": "fˌoʊ pˈɑː", + "final": "fˌoʊ pˈɑː", + "vowel": "fˌoʊ pˈɑː", + "flags": [] + }, + "hors d'oeuvres": { + "mid": "ɔːɹdˈɜːvz", + "final": "ɔːɹdˈɜːvz", + "vowel": "ɔːɹdˈɜːvz", + "flags": [] + }, + "inter alia": { + "mid": "ˌɪntɚɹ ˈeɪliə", + "final": "ˌɪntɚɹ ˈeɪliə", + "vowel": "ˌɪntɚɹ ˈeɪliə", + "flags": [] + }, + "je ne sais": { + "mid": "ʒənəsˈeɪ", + "final": "ʒənəsˈeɪ", + "vowel": "ʒənəsˈeɪ", + "flags": [] + }, + "je t'aime": { + "mid": "ʒətˈɛm", + "final": "ʒətˈɛm", + "vowel": "ʒətˈɛm", + "flags": [] + }, + "la vie": { + "mid": "læ vˈiː", + "final": "læ vˈiː", + "vowel": "læ vˈiː", + "flags": [] + }, + "live in": { + "mid": "lˈɪv ɪn", + "final": "lˈɪv ɪn", + "vowel": "lˈɪv ɪn", + "flags": [] + }, + "main st": { + "mid": "mˈeɪnstɹˌiːt", + "final": "mˈeɪnstɹˌiːt", + "vowel": "mˈeɪnstɹˌiːt", + "flags": [] + }, + "su se": { + "mid": "sˈuːsə", + "final": "sˈuːsə", + "vowel": "sˈuːsə", + "r": "sˈuːsɚ", + "flags": [] + }, + "tae kwon do": { + "mid": "tˈaɪkwɑːndˈoʊ", + "final": "tˈaɪkwɑːndˈoʊ", + "vowel": "tˈaɪkwɑːndˈoʊ", + "flags": [] + }, + "tai chi": { + "mid": "taɪtʃˈiː", + "final": "taɪtʃˈiː", + "vowel": "taɪtʃˈiː", + "flags": [] + }, + "tear apart": { + "mid": "tˌɛɹ ɐpˈɑːɹt", + "final": "tˌɛɹ ɐpˈɑːɹt", + "vowel": "tˌɛɹ ɐpˈɑːɹt", + "flags": [] + }, + "tear off": { + "mid": "tˈɛɹ ˈɑːf", + "final": "tˈɛɹ ˈɑːf", + "vowel": "tˈɛɹ ˈɑːf", + "flags": [] + }, + "van den": { + "mid": "vˈændɛn", + "final": "vˈændɛn", + "vowel": "vˈændɛn", + "flags": [] + }, + "van der": { + "mid": "vˈændɜː", + "final": "vˈændɜː", + "vowel": "vˈændɜːɹ", + "flags": [] + }, + "vis a vis": { + "mid": "vˌiːzɐvˈiː", + "final": "vˌiːzɐvˈiː", + "vowel": "vˌiːzɐvˈiː", + "flags": [] + }, + "wall st": { + "mid": "wˈɔːlstɹˌiːt", + "final": "wˈɔːlstɹˌiːt", + "vowel": "wˈɔːlstɹˌiːt", + "flags": [] + }, + "wear and tear": { + "mid": "wˈɛɹ ɐnd tˈɛɹ", + "final": "wˈɛɹ ɐnd tˈɛɹ", + "vowel": "wˈɛɹ ɐnd tˈɛɹ", + "flags": [] + }, + "wind up": { + "mid": "wˈaɪnd ˈʌp", + "final": "wˈaɪnd ˈʌp", + "vowel": "wˈaɪnd ˈʌp", + "flags": [] + }, + "winds down": { + "mid": "wˈaɪndz dˈaʊn", + "final": "wˈaɪndz dˈaʊn", + "vowel": "wˈaɪndz dˈaʊn", + "flags": [] + }, + "winds up": { + "mid": "wˈaɪndz ˈʌp", + "final": "wˈaɪndz ˈʌp", + "vowel": "wˈaɪndz ˈʌp", + "flags": [] + }, + "wound down": { + "mid": "wˌaʊnd dˈaʊn", + "final": "wˌaʊnd dˈaʊn", + "vowel": "wˌaʊnd dˈaʊn", + "flags": [] + }, + "wound up": { + "mid": "wˌaʊnd ˈʌp", + "final": "wˌaʊnd ˈʌp", + "vowel": "wˌaʊnd ˈʌp", + "flags": [] + }, + "baton rouge": { + "mid": "bˌætən ɹˈuːʒ", + "final": "bˌætən ɹˈuːʒ", + "vowel": "bˌætən ɹˈuːʒ", + "flags": [] + }, + "dehra dun": { + "mid": "dˌɛɹɐdˈuːn", + "final": "dˌɛɹɐdˈuːn", + "vowel": "dˌɛɹɐdˈuːn", + "flags": [] + }, + "des moines": { + "mid": "dəmˈɔɪn", + "final": "dəmˈɔɪn", + "vowel": "dəmˈɔɪn", + "flags": [] + }, + "grand saline": { + "mid": "ɡɹˈændsəlˈiːn", + "final": "ɡɹˈændsəlˈiːn", + "vowel": "ɡɹˈændsəlˈiːn", + "flags": [] + }, + "la jolla": { + "mid": "lɐhˈɔɪə", + "final": "lɐhˈɔɪə", + "vowel": "lɐhˈɔɪə", + "r": "lɐhˈɔɪɚ", + "flags": [] + }, + "la quinta": { + "mid": "ləkˈiːntə", + "final": "ləkˈiːntə", + "vowel": "ləkˈiːntə", + "r": "ləkˈiːntɚ", + "flags": [] + }, + "tel aviv": { + "mid": "tˌɛl ɐvˈiːv", + "final": "tˌɛl ɐvˈiːv", + "vowel": "tˌɛl ɐvˈiːv", + "flags": [] + }, + "le ann": { + "mid": "liːˈæn", + "final": "liːˈæn", + "vowel": "liːˈæn", + "flags": [] + }, + "santa claus": { + "mid": "sˈæntə klˈɔːz", + "final": "sˈæntə klˈɔːz", + "vowel": "sˈæntə klˈɔːz", + "flags": [] + }, + "da vinci": { + "mid": "dɐvˈɪntʃi", + "final": "dɐvˈɪntʃi", + "vowel": "dɐvˈɪntʃi", + "flags": [] + }, + "di maggio": { + "mid": "dᵻmˈɑːʒɪˌoʊ", + "final": "dᵻmˈɑːʒɪˌoʊ", + "vowel": "dᵻmˈɑːʒɪˌoʊ", + "flags": [] + }, + "mc namara": { + "mid": "mˌæknəmˈɑːɹɹə", + "final": "mˌæknəmˈɑːɹɹə", + "vowel": "mˌæknəmˈɑːɹɹə", + "r": "mˌæknəmˈɑːɹɹɚ", + "flags": [] + }, + "each of": { + "mid": "ˈiːtʃ əv", + "final": "ˈiːtʃ əv", + "vowel": "ˈiːtʃ əv", + "flags": [ + "$pause" + ] + }, + "far more": { + "mid": "fˈɑːɹmˌoːɹ", + "final": "fˈɑːɹmˌoːɹ", + "vowel": "fˈɑːɹmˌoːɹ", + "flags": [] + }, + "few more": { + "mid": "fjˈuːmˌoːɹ", + "final": "fjˈuːmˌoːɹ", + "vowel": "fjˈuːmˌoːɹ", + "flags": [] + }, + "here and there": { + "mid": "hˈɪɹ ɐnd ðˈɛɹ", + "final": "hˈɪɹ ɐnd ðˈɛɹ", + "vowel": "hˈɪɹ ɐnd ðˈɛɹ", + "flags": [] + }, + "most of": { + "mid": "mˈoʊst əv", + "final": "mˈoʊst əv", + "vowel": "mˈoʊst əv", + "flags": [] + }, + "such as": { + "mid": "sˈʌtʃ ɐz", + "final": "sˈʌtʃ ɐz", + "vowel": "sˈʌtʃ ɐz", + "flags": [ + "$pause" + ] + }, + "too few": { + "mid": "tˈuː fjuː", + "final": "tˈuː fjuː", + "vowel": "tˈuː fjuː", + "flags": [] + }, + "too many": { + "mid": "tˈuː mɛni", + "final": "tˈuː mɛni", + "vowel": "tˈuː mɛni", + "flags": [] + }, + "too much": { + "mid": "tˈuː mʌtʃ", + "final": "tˈuː mʌtʃ", + "vowel": "tˈuː mʌtʃ", + "flags": [] + }, + "no one": { + "mid": "nˈoʊwˈʌn", + "final": "nˈoʊwˈʌn", + "vowel": "nˈoʊwˈʌn", + "flags": [] + }, + "no longer": { + "mid": "nˌoʊ lˈɑːŋɡɚ", + "final": "nˌoʊ lˈɑːŋɡɚ", + "vowel": "nˌoʊ lˈɑːŋɡɚɹ", + "flags": [] + }, + "no more": { + "mid": "nˈoʊmˌoːɹ", + "final": "nˈoʊmˌoːɹ", + "vowel": "nˈoʊmˌoːɹ", + "flags": [] + }, + "so far": { + "mid": "sˈoʊ fˌɑːɹ", + "final": "sˈoʊ fˌɑːɹ", + "vowel": "sˈoʊ fˌɑːɹ", + "flags": [ + "$strend2" + ] + }, + "so much": { + "mid": "sˈoʊ mˌʌtʃ", + "final": "sˈoʊ mˌʌtʃ", + "vowel": "sˈoʊ mˌʌtʃ", + "flags": [] + }, + "of a": { + "mid": "əvə", + "final": "əvə", + "vowel": "əvə", + "r": "əvɚ", + "flags": [ + "$nounf" + ] + }, + "of an": { + "mid": "əvən", + "final": "əvən", + "vowel": "əvən", + "flags": [ + "$nounf" + ] + }, + "of which": { + "mid": "ʌvwˈɪtʃ", + "final": "ʌvwˈɪtʃ", + "vowel": "ʌvwˈɪtʃ", + "flags": [ + "$2", + "$pause" + ] + }, + "of the": { + "mid": "ʌvðə", + "final": "ʌvðə", + "vowel": "ʌvðɪ", + "flags": [ + "$nounf" + ] + }, + "for a": { + "mid": "fɚɹə", + "final": "fɚɹə", + "vowel": "fɚɹə", + "r": "fɚɹɚ", + "flags": [ + "$nounf" + ] + }, + "for an": { + "mid": "fɚɹən", + "final": "fɚɹən", + "vowel": "fɚɹən", + "flags": [ + "$nounf" + ] + }, + "for the": { + "mid": "fɚðə", + "final": "fɚðə", + "vowel": "fɚðɪ", + "flags": [ + "$nounf" + ] + }, + "for a while": { + "mid": "fɚɹə wˈaɪl", + "final": "fɚɹə wˈaɪl", + "vowel": "fɚɹə wˈaɪl", + "flags": [] + }, + "for one": { + "final": "fɔːɹwˈʌn", + "flags": [ + "$2", + "$atend" + ] + }, + "to be": { + "mid": "təbi", + "vowel": "təbi", + "flags": [ + "$atend", + "$pastf" + ] + }, + "to to": { + "mid": "tʊtə", + "final": "tʊtʊ", + "vowel": "tʊtʊ", + "flags": [ + "$verbf" + ] + }, + "to and fro": { + "mid": "tˌuːəndfɹˈoʊ", + "final": "tˌuːəndfɹˈoʊ", + "vowel": "tˌuːəndfɹˈoʊ", + "flags": [] + }, + "at a": { + "mid": "æɾə", + "final": "æɾə", + "vowel": "æɾə", + "r": "æɾɚ", + "flags": [ + "$nounf" + ] + }, + "at it": { + "mid": "æɾ ɪt", + "final": "ˈæɾɪt", + "vowel": "æɾ ɪɾ", + "flags": [ + "$atend" + ] + }, + "at once": { + "mid": "ɐtwˈʌns", + "final": "ɐtwˈʌns", + "vowel": "ɐtwˈʌns", + "flags": [] + }, + "at will": { + "mid": "ɐtwˈɪl", + "final": "ɐtwˈɪl", + "vowel": "ɐtwˈɪl", + "flags": [] + }, + "with the": { + "mid": "wɪððə", + "final": "wɪððə", + "vowel": "wɪððɪ", + "flags": [ + "$nounf" + ] + }, + "in the": { + "mid": "ɪnðə", + "final": "ɪnðə", + "vowel": "ɪnðɪ", + "flags": [] + }, + "in which": { + "mid": "ɪnwˌɪtʃ", + "final": "ɪnwˌɪtʃ", + "vowel": "ɪnwˌɪtʃ", + "flags": [ + "$pause", + "$u2" + ] + }, + "in a row": { + "mid": "ˌɪnɐ ɹˈoʊ", + "final": "ˌɪnɐ ɹˈoʊ", + "vowel": "ˌɪnɐ ɹˈoʊ", + "flags": [] + }, + "on the": { + "mid": "ɔnðə", + "final": "ɔnðə", + "vowel": "ɔnðɪ", + "flags": [] + }, + "out of": { + "mid": "ˌaʊɾəv", + "final": "ˌaʊɾəv", + "vowel": "ˌaʊɾəv", + "flags": [] + }, + "from the": { + "mid": "fɹʌmðə", + "final": "fɹʌmðə", + "vowel": "fɹʌmðɪ", + "flags": [] + }, + "from where": { + "mid": "fɹʌm wˈɛɹ", + "vowel": "fɹʌm wˈɛɹ", + "flags": [ + "$pause" + ] + }, + "from which": { + "mid": "fɹʌm wˈɪtʃ", + "vowel": "fɹʌm wˈɪtʃ", + "flags": [ + "$pause" + ] + }, + "per cent": { + "mid": "pɚ sˈɛnt", + "final": "pɚ sˈɛnt", + "vowel": "pɚ sˈɛnt", + "flags": [] + }, + "per se": { + "mid": "pˌɜː sˈeɪ", + "final": "pˌɜː sˈeɪ", + "vowel": "pˌɜː sˈeɪ", + "flags": [] + }, + "was a": { + "mid": "wʌzɐ", + "final": "wʌzɐ", + "vowel": "wʌzɐ", + "flags": [] + }, + "was the": { + "mid": "wʌzðə", + "final": "wʌzðə", + "vowel": "wʌzðɪ", + "flags": [] + }, + "will to": { + "mid": "wˈɪltə", + "final": "wˈɪltʊ", + "vowel": "wˈɪltʊ", + "flags": [] + }, + "would have": { + "mid": "wʊdhɐv", + "final": "wˈʊdhæv", + "vowel": "wʊdhɐv", + "flags": [ + "$atend", + "$pastf" + ] + }, + "would have to": { + "mid": "wʊdhˌævtə", + "final": "wʊdhˈævtʊ", + "vowel": "wʊdhˌævtʊ", + "flags": [ + "$strend2", + "$u2", + "$verbf" + ] + }, + "wouldn't have": { + "mid": "wˌʊdəntəv", + "final": "wˈʊdəntəv", + "vowel": "wˌʊdəntəv", + "flags": [ + "$pastf", + "$strend2", + "$u1" + ] + }, + "wouldn't have to": { + "mid": "wˌʊdəntævtə", + "final": "wˈʊdəntævtʊ", + "vowel": "wˌʊdəntævtʊ", + "flags": [ + "$strend2", + "$verbf" + ] + }, + "won't have": { + "mid": "wˌoʊntɐv", + "final": "wˈoʊnthæv", + "vowel": "wˌoʊntɐv", + "flags": [ + "$atend", + "$pastf", + "$u1" + ] + }, + "won't have it": { + "mid": "woʊnthˌævɪt", + "final": "woʊnthˈævɪt", + "vowel": "woʊnthˌævɪt", + "flags": [ + "$u2+" + ] + }, + "won't have to": { + "mid": "wˈoʊntævtə", + "final": "wˈoʊntævtʊ", + "vowel": "wˈoʊntævtʊ", + "flags": [ + "$strend2", + "$verbf" + ] + }, + "must have": { + "mid": "mˈʌstɐv", + "final": "mˈʌsthæv", + "vowel": "mˈʌstɐv", + "flags": [ + "$1", + "$atend", + "$pastf" + ] + }, + "must have to": { + "mid": "mˌʌstævtə", + "final": "mˌʌstævtʊ", + "vowel": "mˌʌstævtʊ", + "flags": [ + "$pastf" + ] + }, + "mustn't have": { + "mid": "mˌʌsənthɐv", + "final": "mˈʌsənthɐv", + "vowel": "mˌʌsənthɐv", + "flags": [ + "$pastf", + "$u1+" + ] + }, + "mustn't have to": { + "mid": "mˈʌsənthævtə", + "final": "mˈʌsənthævtʊ", + "vowel": "mˈʌsənthævtʊ", + "flags": [ + "$pastf" + ] + }, + "should have": { + "mid": "ʃˌʊdəv", + "final": "ʃˈʊdhæv", + "vowel": "ʃˌʊdəv", + "flags": [ + "$atend", + "$pastf", + "$u1+" + ] + }, + "should have it": { + "mid": "ʃʊdhˌævɪt", + "final": "ʃʊdhˈævɪt", + "vowel": "ʃʊdhˌævɪt", + "flags": [ + "$u2+" + ] + }, + "should have to": { + "mid": "ʃʊdhˌævtə", + "final": "ʃʊdhˈævtʊ", + "vowel": "ʃʊdhˌævtʊ", + "flags": [ + "$strend2", + "$u2", + "$verbf" + ] + }, + "shouldn't have": { + "mid": "ʃˌʊdənthəv", + "final": "ʃˈʊdənthəv", + "vowel": "ʃˌʊdənthəv", + "flags": [ + "$pastf", + "$u1+" + ] + }, + "shouldn't have to": { + "mid": "ʃˈʊdəntævtə", + "final": "ʃˈʊdəntævtʊ", + "vowel": "ʃˈʊdəntævtʊ", + "flags": [ + "$strend2", + "$verbf" + ] + }, + "can't have": { + "mid": "kˈæntɐv", + "final": "kˈæntɐv", + "vowel": "kˈæntɐv", + "flags": [ + "$pastf" + ] + }, + "can't have to": { + "mid": "kˈæntɐv tə", + "final": "kˌæntˈævtʊ", + "vowel": "kˈæntɐv tʊ", + "flags": [ + "$atend", + "$verbf" + ] + }, + "could have": { + "mid": "kˌʊdɐv", + "final": "kˈʊdhæv", + "vowel": "kˌʊdɐv", + "flags": [ + "$atend", + "$pastf" + ] + }, + "could have to": { + "mid": "kʊdhˌævtə", + "final": "kʊdhˈævtʊ", + "vowel": "kʊdhˌævtʊ", + "flags": [ + "$strend2", + "$u2", + "$verbf" + ] + }, + "could have it": { + "mid": "kʊdhˈævɪt", + "final": "kʊdhˈævɪt", + "vowel": "kʊdhˈævɪt", + "flags": [] + }, + "couldn't have": { + "mid": "kˌʊdəntəv", + "final": "kˈʊdəntəv", + "vowel": "kˌʊdəntəv", + "flags": [ + "$pastf", + "$u1+" + ] + }, + "couldn't have to": { + "mid": "kˈʊdəntævtə", + "final": "kˈʊdəntævtʊ", + "vowel": "kˈʊdəntævtʊ", + "flags": [ + "$strend2", + "$verbf" + ] + }, + "may have": { + "mid": "mˌeɪhɐv", + "final": "mˈeɪhɐv", + "vowel": "mˌeɪhɐv", + "flags": [ + "$pastf", + "$u1+" + ] + }, + "may have to": { + "mid": "mˌeɪhɐv tə", + "final": "mˌeɪhˈævtʊ", + "vowel": "mˌeɪhɐv tʊ", + "flags": [ + "$atend", + "$verbf" + ] + }, + "might have": { + "mid": "mˌaɪthɐv", + "final": "mˈaɪthɐv", + "vowel": "mˌaɪthɐv", + "flags": [ + "$pastf", + "$u1+" + ] + }, + "might have to": { + "mid": "mˌaɪthɐv tə", + "final": "mˌaɪthˈævtʊ", + "vowel": "mˌaɪthɐv tʊ", + "flags": [ + "$atend", + "$verbf" + ] + }, + "have been": { + "mid": "hɐvbɪn", + "final": "hɐvbɪn", + "vowel": "hɐvbɪn", + "flags": [ + "$pastf" + ] + }, + "have not": { + "mid": "hɐvnˌɑːt", + "final": "hɐvnˈɑːt", + "vowel": "hɐvnˌɑːt", + "flags": [ + "$atend", + "$pastf" + ] + }, + "has been": { + "mid": "hˈæzbiːn", + "final": "hˈæzbiːn", + "vowel": "hˈæzbiːn", + "flags": [ + "$pastf" + ] + }, + "has to": { + "final": "hˈæztuː", + "flags": [ + "$atend" + ] + }, + "has not": { + "mid": "həznɑːt", + "final": "hɐznˈɑːt", + "vowel": "həznɑːt", + "flags": [ + "$atend", + "$pastf" + ] + }, + "has it": { + "final": "hˌæzɪt", + "flags": [ + "$atend" + ] + }, + "had to": { + "mid": "hædtə", + "final": "hˈædtuː", + "vowel": "hædtʊ", + "flags": [ + "$atend" + ] + }, + "had been": { + "mid": "hɐdbɪn", + "final": "hɐdbɪn", + "vowel": "hɐdbɪn", + "flags": [ + "$pastf" + ] + }, + "had it": { + "mid": "hˌædɪt", + "final": "hˌædɪt", + "vowel": "hˌædɪt", + "flags": [] + }, + "had them": { + "mid": "hˌædðɛm", + "final": "hˌædðɛm", + "vowel": "hˌædðɛm", + "flags": [] + }, + "had one": { + "final": "hˈædwʌn", + "flags": [ + "$atend" + ] + }, + "had any": { + "mid": "hˌæd ˌɛni", + "final": "hˌæd ˌɛni", + "vowel": "hˌæd ˌɛni", + "flags": [] + }, + "had some": { + "mid": "hˌæd sʌm", + "final": "hˌæd sʌm", + "vowel": "hˌæd sʌm", + "flags": [] + }, + "do not": { + "mid": "duːnˌɑːt", + "final": "duːnˈɑːt", + "vowel": "duːnˌɑːt", + "flags": [ + "$u2+", + "$verbf" + ] + }, + "does not": { + "mid": "dʌznˌɑːt", + "final": "dʌznˈɑːt", + "vowel": "dʌznˌɑːt", + "flags": [ + "$u2+", + "$verbf" + ] + }, + "did not": { + "mid": "dɪdnˌɑːt", + "final": "dɪdnˈɑːt", + "vowel": "dɪdnˌɑːt", + "flags": [ + "$u2+", + "$verbf" + ] + }, + "that's it": { + "mid": "ðætsˈɪt", + "final": "ðætsˈɪt", + "vowel": "ðætsˈɪt", + "flags": [] + }, + "i've had": { + "mid": "aɪvhˌæd", + "final": "aɪvhˈæd", + "vowel": "aɪvhˌæd", + "flags": [ + "$u2+" + ] + }, + "you've had": { + "mid": "juːvhˌæd", + "final": "juːvhˈæd", + "vowel": "juːvhˌæd", + "flags": [ + "$u2+" + ] + }, + "he's had": { + "mid": "hiːzhˌæd", + "final": "hiːzhˈæd", + "vowel": "hiːzhˌæd", + "flags": [ + "$u2+" + ] + }, + "she's had": { + "mid": "ʃiːzhˌæd", + "final": "ʃiːzhˈæd", + "vowel": "ʃiːzhˌæd", + "flags": [ + "$u2+" + ] + }, + "it's had": { + "mid": "ɪtshˌæd", + "final": "ɪtshˈæd", + "vowel": "ɪtshˌæd", + "flags": [ + "$u2+" + ] + }, + "we've had": { + "mid": "wiːvhˌæd", + "final": "wiːvhˈæd", + "vowel": "wiːvhˌæd", + "flags": [ + "$u2+" + ] + }, + "they've had": { + "mid": "ðeɪvhˌæd", + "final": "ðeɪvhˈæd", + "vowel": "ðeɪvhˌæd", + "flags": [ + "$u2+" + ] + }, + "i am": { + "midcap": "aɪɐm", + "startcap": "aɪɐm", + "flags": [ + "$atend" + ] + }, + "i shall": { + "midcap": "aɪʃˌɐl", + "startcap": "aɪʃˌɐl", + "flags": [ + "$atend" + ] + }, + "it is": { + "mid": "ɪɾ ɪz", + "final": "ɪɾ ˈɪz", + "vowel": "ɪɾ ɪz", + "flags": [ + "$atend" + ] + }, + "it was": { + "final": "ɪt wˈʌz", + "flags": [ + "$atend" + ] + }, + "we shall": { + "mid": "wiːʃˌɐl", + "final": "wiːʃˈæl", + "vowel": "wiːʃˌɐl", + "flags": [ + "$atend" + ] + }, + "i had": { + "final": "aɪ hˈæd", + "flags": [ + "$atend" + ] + }, + "he had": { + "final": "hiː hˈæd", + "flags": [ + "$atend" + ] + }, + "as is": { + "mid": "ˌæzˌɪz", + "final": "ˌæzˌɪz", + "vowel": "ˌæzˌɪz", + "flags": [ + "$pause" + ] + }, + "as it is": { + "mid": "ˌæzɪtˌɪz", + "final": "ˌæzɪtˈɪz", + "vowel": "ˌæzɪtˌɪz", + "flags": [ + "$u+" + ] + }, + "do so": { + "final": "dˈuː soʊ", + "flags": [ + "$atend" + ] + }, + "not have": { + "mid": "nˌɑːɾɐv", + "final": "nˈɑːthæv", + "vowel": "nˌɑːɾɐv", + "flags": [ + "$atend", + "$pastf", + "$u1" + ] + }, + "not have to": { + "mid": "nˌɑːthævtə", + "final": "nˈɑːthævtʊ", + "vowel": "nˌɑːthævtʊ", + "flags": [ + "$pastf", + "$strend" + ] + }, + "not a": { + "mid": "nˌɑːɾə", + "final": "nˌɑːɾə", + "vowel": "nˌɑːɾə", + "r": "nˌɑːɾɚ", + "flags": [ + "$nounf" + ] + }, + "many of": { + "mid": "mˈɛnɪəv", + "final": "mˈɛnɪəv", + "vowel": "mˈɛnɪəv", + "flags": [] + }, + "some one": { + "mid": "sˈʌmwʌn", + "final": "sˈʌmwʌn", + "vowel": "sˈʌmwʌn", + "flags": [] + }, + "this one": { + "mid": "ðˈɪswˌʌn", + "final": "ðˈɪswˌʌn", + "vowel": "ðˈɪswˌʌn", + "flags": [ + "$verbsf" + ] + }, + "that a": { + "mid": "ðˌæɾə", + "final": "ðˌæɾə", + "vowel": "ðˌæɾə", + "r": "ðˌæɾɚ", + "flags": [ + "$nounf" + ] + }, + "that it": { + "mid": "ðˌɐɾɪt", + "final": "ðætˈɪt", + "vowel": "ðˌɐɾɪt", + "flags": [ + "$atend", + "$verbsf" + ] + }, + "that is": { + "final": "ðæt ˈɪz", + "flags": [ + "$atend" + ] + }, + "that was": { + "final": "ðɐt wˈʌz", + "flags": [ + "$atend" + ] + }, + "that one": { + "mid": "ðˈætwˌʌn", + "final": "ðˈætwˌʌn", + "vowel": "ðˈætwˌʌn", + "flags": [] + }, + "that the": { + "mid": "ðætðə", + "final": "ðætðə", + "vowel": "ðætðɪ", + "flags": [] + }, + "that has been": { + "mid": "ðɐthɐzbˌɪn", + "final": "ðɐthɐzbˈɪn", + "vowel": "ðɐthɐzbˌɪn", + "flags": [ + "$u+" + ] + }, + "that's been": { + "mid": "ðɐtsbˌɪn", + "final": "ðɐtsbˈɪn", + "vowel": "ðɐtsbˌɪn", + "flags": [ + "$u+" + ] + }, + "there are": { + "mid": "ðɛɹˌɑːɹ", + "final": "ðɛɹˈɑːɹ", + "vowel": "ðɛɹˌɑːɹ", + "flags": [ + "$strend" + ] + }, + "there is": { + "final": "ðɛɹˈɪz", + "flags": [ + "$atend" + ] + }, + "there be": { + "mid": "ðɛɹbˈiː", + "final": "ðɛɹbˈiː", + "vowel": "ðɛɹbˈiː", + "flags": [] + }, + "there was": { + "mid": "ðɛɹwˌʌz", + "final": "ðɛɹwˈʌz", + "vowel": "ðɛɹwˌʌz", + "flags": [ + "$strend" + ] + }, + "there were": { + "mid": "ðɛɹwˌɜː", + "final": "ðɛɹwˈɜː", + "vowel": "ðɛɹwˌɜːɹ", + "flags": [ + "$strend" + ] + }, + "than a": { + "mid": "ðˌænə", + "final": "ðˌænə", + "vowel": "ðˌænə", + "r": "ðˌænɚ", + "flags": [ + "$nounf" + ] + }, + "than an": { + "mid": "ðˌænən", + "final": "ðˌænən", + "vowel": "ðˌænən", + "flags": [ + "$nounf" + ] + }, + "a la": { + "mid": "ˌɑːlæ", + "final": "ˌɑːlæ", + "vowel": "ˌɑːlæ", + "flags": [] + }, + "a while": { + "mid": "ɐ wˈaɪl", + "vowel": "ɐ wˈaɪl", + "flags": [] + }, + "the while": { + "mid": "ðə wˈaɪl", + "vowel": "ðə wˈaɪl", + "flags": [] + } + }, + "homographs": { + "absent": { + "verb": "æbsˈɛnt" + }, + "absently": { + "verb": "æbsˈɛntli" + }, + "absentness": { + "verb": "æbsˈɛntnəs" + }, + "abstract": { + "verb": "ɐbstɹˈækt" + }, + "abuse": { + "verb": "ɐbjˈuːz" + }, + "abusedly": { + "verb": "ɐbjˈuːzᵻdli" + }, + "advert": { + "verb": "ɐdvˈɜːt" + }, + "alternate": { + "verb": "ˈɔːltɚnˌeɪt" + }, + "am": { + "verb": "ɐm", + "noun": "ɐm", + "past": "ɐm" + }, + "analyses": { + "noun": "ɐnˈæləsˌiːz" + }, + "appropriate": { + "verb": "ɐpɹˈoʊpɹɪˌeɪt" + }, + "appropriately": { + "verb": "ɐpɹˈoʊpɹɪˌeɪtli" + }, + "appropriateness": { + "verb": "ɐpɹˈoʊpɹɪˌeɪtnəs" + }, + "approximate": { + "verb": "ɐpɹˈɑːksɪmˌeɪt" + }, + "attribute": { + "verb": "ɐtɹˈɪbjuːt" + }, + "bow": { + "verb": "bˈaʊ" + }, + "bowhead": { + "verb": "bˈaʊhɛd" + }, + "bowheads": { + "verb": "bˈaʊhɛdz" + }, + "bowless": { + "verb": "bˈaʊləs" + }, + "bowly": { + "verb": "bˈaʊli" + }, + "breath": { + "verb": "bɹˈiːð" + }, + "buffet": { + "noun": "bˈʌfeɪ" + }, + "buffeted": { + "noun": "bˈʌfeɪᵻd" + }, + "buffeting": { + "noun": "bˈʌfeɪɪŋ" + }, + "buffetings": { + "noun": "bˈʌfeɪɪŋz" + }, + "buffets": { + "noun": "bˈʌfeɪs" + }, + "close": { + "verb": "klˈoʊz" + }, + "cloth": { + "verb": "klˈoʊð" + }, + "compound": { + "verb": "kɑːmpˈaʊnd" + }, + "concert": { + "verb": "kənsˈɜːt" + }, + "conduct": { + "verb": "kəndˈʌkt" + }, + "conflict": { + "verb": "kənflˈɪkt" + }, + "conscript": { + "verb": "kɑːnskɹˈɪpt" + }, + "console": { + "verb": "kənsˈoʊl" + }, + "consolement": { + "verb": "kənsˈoʊlmənt" + }, + "consort": { + "verb": "kənsˈɔːɹt" + }, + "construct": { + "verb": "kənstɹˈʌkt" + }, + "content": { + "verb": "kəntˈɛnt", + "past": "kəntˈɛnt" + }, + "contents": { + "past": "kəntˈɛnts" + }, + "contest": { + "verb": "kəntˈɛst" + }, + "contract": { + "verb": "kəntɹˈækt" + }, + "contrast": { + "verb": "kəntɹˈæst" + }, + "converse": { + "verb": "kənvˈɜːs" + }, + "convert": { + "noun": "kˈɑːnvɜːt" + }, + "converts": { + "noun": "kˈɑːnvɜːts" + }, + "convict": { + "verb": "kənvˈɪkt" + }, + "coordinate": { + "verb": "koʊˈɔːɹdɪnˌeɪt" + }, + "coordinately": { + "verb": "koʊˈɔːɹdɪnˌeɪtli" + }, + "defect": { + "verb": "dᵻfˈɛkt" + }, + "degenerate": { + "noun": "dᵻdʒˈɛnɚɹət" + }, + "degenerates": { + "noun": "dᵻdʒˈɛnɚɹəts" + }, + "deliberate": { + "verb": "dᵻlˈɪbɚɹˌeɪt" + }, + "deliberately": { + "verb": "dᵻlˈɪbɚɹˌeɪtli" + }, + "deliberateness": { + "verb": "dᵻlˈɪbɚɹˌeɪtnəs" + }, + "desert": { + "verb": "dᵻzˈɜːt" + }, + "digest": { + "noun": "dˈaɪdʒɛst" + }, + "digests": { + "noun": "dˈaɪdʒɛsts" + }, + "disabuse": { + "verb": "dˌɪsɐbjˈuːz" + }, + "disappropriate": { + "verb": "dɪsɐpɹˈoʊpɹɪˌeɪt" + }, + "discontents": { + "past": "dɪskəntˈɛnts" + }, + "dove": { + "verb": "dˈoʊv", + "past": "dˈoʊv" + }, + "dovelike": { + "verb": "dˈoʊvlaɪk", + "past": "dˈoʊvlaɪk" + }, + "doves": { + "past": "dˈoʊvz" + }, + "egress": { + "verb": "ɪɡɹˈɛs" + }, + "elaborate": { + "noun": "ᵻlˈæbɚɹət" + }, + "ennis": { + "verb": "əniz", + "noun": "əniz", + "past": "əniz" + }, + "entrance": { + "verb": "ɛntɹˈæns" + }, + "entrancement": { + "verb": "ɛntɹˈænsmənt" + }, + "entrancements": { + "verb": "ɛntɹˈænsmənts" + }, + "envelope": { + "verb": "ɛnvˈɛləp" + }, + "escort": { + "verb": "ɛskˈɔːɹt" + }, + "estimate": { + "verb": "ˈɛstᵻmˌeɪt" + }, + "exploit": { + "verb": "ɛksplˈɔɪt" + }, + "export": { + "verb": "ɛkspˈoːɹt" + }, + "extract": { + "verb": "ɛkstɹˈækt" + }, + "filtrate": { + "verb": "fɪltɹˈeɪt" + }, + "finance": { + "verb": "faɪnˈæns" + }, + "fragment": { + "verb": "fɹæɡmˈɛnt" + }, + "graduate": { + "verb": "ɡɹˈædjuːˌeɪt" + }, + "house": { + "verb": "hˈaʊz" + }, + "houseful": { + "verb": "hˈaʊzfəl" + }, + "housefuls": { + "verb": "hˈaʊzfəlz" + }, + "household": { + "verb": "hˈaʊzhoʊld" + }, + "households": { + "verb": "hˈaʊzhoʊldz" + }, + "houseless": { + "verb": "hˈaʊzləs" + }, + "houselessness": { + "verb": "hˈaʊzləsnəs" + }, + "housework": { + "verb": "hˈaʊzwɜːk" + }, + "hydrate": { + "verb": "haɪdɹˈeɪt" + }, + "implant": { + "verb": "ɪmplˈænt" + }, + "incense": { + "verb": "ɪnsˈɛns" + }, + "increase": { + "verb": "ɪŋkɹˈiːs" + }, + "inseparate": { + "verb": "ɪnsˈɛpɚɹˌeɪt" + }, + "insert": { + "verb": "ɪnsˈɜːt" + }, + "internment": { + "noun": "ˈɪntɜːnmənt" + }, + "internments": { + "noun": "ˈɪntɜːnmənts" + }, + "internship": { + "noun": "ˈɪntɜːnʃˌɪp" + }, + "internships": { + "noun": "ˈɪntɜːnʃˌɪps" + }, + "interrupt": { + "verb": "ˌɪntɚɹˈʌpt" + }, + "interwind": { + "verb": "ˌɪntɚwˈaɪnd" + }, + "inwind": { + "verb": "ɪnwˈaɪnd" + }, + "lactate": { + "verb": "læktˈeɪt" + }, + "learned": { + "noun": "lˈɜːnᵻd" + }, + "learnedness": { + "noun": "lˈɜːnᵻdnəs" + }, + "live": { + "verb": "lˈɪv" + }, + "liveness": { + "verb": "lˈɪvnəs" + }, + "mandate": { + "verb": "mændˈeɪt" + }, + "misappropriate": { + "verb": "mɪsɐpɹˈoʊpɹɪˌeɪt" + }, + "misconduct": { + "verb": "mɪskəndˈʌkt" + }, + "misestimate": { + "verb": "mɪsˈɛstᵻmˌeɪt" + }, + "misuse": { + "verb": "mɪsjˈuːz" + }, + "moderate": { + "verb": "mˈɑːdɚɹˌeɪt" + }, + "moderately": { + "verb": "mˈɑːdɚɹˌeɪtli" + }, + "object": { + "verb": "ɑːbdʒˈɛkt" + }, + "overestimate": { + "verb": "ˌoʊvɚɹˈɛstᵻmˌeɪt" + }, + "overlearned": { + "noun": "ˌoʊvɚlˈɜːnᵻd" + }, + "overlive": { + "verb": "ˌoʊvɚlˈɪv" + }, + "overuse": { + "verb": "ˌoʊvɚjˈuːz" + }, + "overwind": { + "verb": "ˌoʊvɚwˈaɪnd" + }, + "perfect": { + "verb": "pɚfˈɛkt" + }, + "permit": { + "verb": "pɚmˈɪt" + }, + "pervert": { + "noun": "pˈɜːvɚt" + }, + "pervertedly": { + "noun": "pˈɜːvɚɾᵻdli" + }, + "perverts": { + "noun": "pˈɜːvɚts" + }, + "predigest": { + "noun": "pɹiːdˈaɪdʒɛst" + }, + "predigests": { + "noun": "pɹiːdˈaɪdʒɛsts" + }, + "present": { + "verb": "pɹɪzˈɛnt" + }, + "produce": { + "noun": "pɹˈɑːduːs" + }, + "progress": { + "verb": "pɹəɡɹˈɛs" + }, + "project": { + "verb": "pɹədʒˈɛkt" + }, + "prostrate": { + "verb": "pɹɑːstɹˈeɪt" + }, + "protest": { + "verb": "pɹətˈɛst" + }, + "read": { + "past": "ɹˈɛd" + }, + "reappropriate": { + "verb": "ɹˌiːɐpɹˈoʊpɹɪˌeɪt" + }, + "rebel": { + "verb": "ɹᵻbˈɛl" + }, + "recall": { + "verb": "ɹᵻkˈɔːl" + }, + "reconduct": { + "verb": "ɹˌiːkəndˈʌkt" + }, + "reconvert": { + "noun": "ɹˌiːkˈɑːnvɜːt" + }, + "reconverts": { + "noun": "ɹˌiːkˈɑːnvɜːts" + }, + "reconvict": { + "verb": "ɹˌiːkənvˈɪkt" + }, + "record": { + "verb": "ɹᵻkˈoːɹd" + }, + "reentrance": { + "verb": "ɹiːɛntɹˈæns" + }, + "refund": { + "noun": "ɹˈiːfʌnd" + }, + "refunds": { + "noun": "ɹˈiːfʌndz" + }, + "refuse": { + "noun": "ɹˈɛfjuːs" + }, + "reimplant": { + "verb": "ɹˌiːɪmplˈænt" + }, + "remit": { + "verb": "ɹiːmˈɪt" + }, + "remitter": { + "verb": "ɹiːmˈɪɾɚ" + }, + "resent": { + "past": "ɹiːsˈɛnt" + }, + "resented": { + "past": "ɹiːsˈɛntᵻd" + }, + "resentful": { + "past": "ɹiːsˈɛntfəl" + }, + "resentfulness": { + "past": "ɹiːsˈɛntfəlnəs" + }, + "resenting": { + "past": "ɹiːsˈɛntɪŋ" + }, + "resentment": { + "past": "ɹiːsˈɛntmənt" + }, + "resentments": { + "past": "ɹiːsˈɛntmənts" + }, + "resents": { + "past": "ɹiːsˈɛnts" + }, + "separate": { + "verb": "sˈɛpɚɹˌeɪt" + }, + "separately": { + "verb": "sˈɛpɚɹˌeɪtli" + }, + "separateness": { + "verb": "sˈɛpɚɹˌeɪtnəs" + }, + "slaver": { + "verb": "slˈævɚ" + }, + "slough": { + "verb": "slˈʌf" + }, + "some": { + "past": "sʌm" + }, + "subordinate": { + "verb": "sʌbˈoːɹdᵻnˌeɪt" + }, + "survey": { + "verb": "sɚvˈeɪ" + }, + "suspect": { + "verb": "səspˈɛkt" + }, + "tear": { + "verb": "tˈɛɹ" + }, + "tearful": { + "verb": "tˈɛɹfəl" + }, + "tearfulness": { + "verb": "tˈɛɹfəlnəs" + }, + "tearless": { + "verb": "tˈɛɹləs" + }, + "torment": { + "verb": "toːɹmˈɛnt" + }, + "tormentful": { + "verb": "toːɹmˈɛntfəl" + }, + "transf": { + "verb": "tɹɐnsf", + "noun": "tɹɐnsf", + "past": "tɹɐnsf" + }, + "transfer": { + "verb": "tɹænsfˈɜː" + }, + "transl": { + "verb": "tɹɐnsəl", + "noun": "tɹɐnsəl", + "past": "tɹɐnsəl" + }, + "transp": { + "verb": "tɹɐnsp", + "noun": "tɹɐnsp", + "past": "tɹɐnsp" + }, + "transport": { + "verb": "tɹænspˈoːɹt" + }, + "triplicate": { + "verb": "tɹˈɪplᵻkˌeɪt" + }, + "unappropriate": { + "verb": "ʌnɐpɹˈoʊpɹɪˌeɪt" + }, + "unbow": { + "verb": "ʌnbˈaʊ" + }, + "unconvert": { + "noun": "ʌŋkˈɑːnvɜːt" + }, + "undeliberate": { + "verb": "ˌʌndᵻlˈɪbɚɹˌeɪt" + }, + "undercloth": { + "verb": "ˌʌndɚklˈoʊð" + }, + "underestimate": { + "verb": "ˌʌndɚɹˈɛstᵻmˌeɪt" + }, + "undergraduate": { + "verb": "ˌʌndɚɡɹˈædjuːˌeɪt" + }, + "underground": { + "noun": "ˈʌndɚɡɹˌaʊnd" + }, + "undergrounds": { + "noun": "ˈʌndɚɡɹˌaʊndz" + }, + "unlearned": { + "noun": "ʌnlˈɜːnᵻd" + }, + "unlive": { + "verb": "ʌnlˈɪv" + }, + "unn": { + "verb": "ənn", + "noun": "ənn", + "past": "ənn" + }, + "unresented": { + "past": "ˌʌnɹiːsˈɛntᵻd" + }, + "unseparate": { + "verb": "ʌnsˈɛpɚɹˌeɪt" + }, + "unuse": { + "verb": "ʌnjˈuːz" + }, + "upbring": { + "verb": "əpbɹɪŋ", + "noun": "əpbɹɪŋ", + "past": "əpbɹɪŋ" + }, + "upbringing": { + "verb": "əpbɹɪŋɪŋ", + "noun": "əpbɹɪŋɪŋ", + "past": "əpbɹɪŋɪŋ" + }, + "update": { + "verb": "ʌpdˈeɪt" + }, + "upgrade": { + "verb": "ʌpɡɹˈeɪd" + }, + "upset": { + "noun": "ˈʌpsɛt" + }, + "upsetment": { + "noun": "ˈʌpsɛtmənt" + }, + "upsets": { + "noun": "ˈʌpsɛts" + }, + "upsetters": { + "noun": "ˈʌpsɛɾɚz" + }, + "use": { + "verb": "jˈuːz" + }, + "usehold": { + "verb": "jˈuːzhoʊld" + }, + "useless": { + "verb": "jˈuːzləs" + }, + "uselessly": { + "verb": "jˈuːzləsli" + }, + "uselessness": { + "verb": "jˈuːzləsnəs" + }, + "while": { + "noun": "wˈaɪl" + }, + "wind": { + "verb": "wˈaɪnd" + }, + "windless": { + "verb": "wˈaɪndləs" + }, + "windlike": { + "verb": "wˈaɪndlaɪk" + }, + "wound": { + "past": "wˈaʊnd" + } + }, + "flag_sets": { + "verbf": [ + "absolutely", + "already", + "alternately", + "amply", + "busily", + "can", + "can't", + "closely", + "could", + "couldn't", + "did", + "didn't", + "do", + "does", + "doesn't", + "doesnt", + "don't", + "he'll", + "i", + "i'll", + "just", + "may", + "might", + "must", + "mustn't", + "never", + "nobly", + "now", + "preferably", + "presently", + "severely", + "shall", + "shalln't", + "shalt", + "shan't", + "she'll", + "should", + "shouldn't", + "shrilly", + "singly", + "soon", + "still", + "subtly", + "then", + "there'll", + "they", + "they'll", + "to", + "truely", + "we", + "we'll", + "which", + "who", + "will", + "won't", + "would", + "you", + "you'll" + ], + "verbsf": [ + "he", + "it", + "she", + "that", + "this" + ], + "nounf": [ + "a", + "an", + "another", + "any", + "at", + "every", + "her", + "his", + "in", + "its", + "many", + "my", + "one", + "our", + "some", + "that", + "the", + "their", + "these", + "this", + "those", + "transports", + "your" + ], + "pastf": [ + "are", + "aren't", + "be", + "been", + "being", + "get", + "getting", + "got", + "had", + "hadn't", + "has", + "hasn't", + "hath", + "have", + "haven't", + "havent", + "having", + "he's", + "i've", + "is", + "isn't", + "she's", + "they've", + "was", + "wasn't", + "wast", + "we'd", + "we've", + "were", + "weren't", + "you've" + ], + "verbextend": [ + "not", + "only" + ], + "pause": [ + "although", + "and", + "because", + "but", + "despite", + "furthermore", + "ie", + "if", + "instead", + "nor", + "once", + "or", + "regardless", + "since", + "thus", + "whatever", + "whenever", + "where", + "which", + "whilst", + "who", + "whose" + ], + "brk": [ + "while", + "within" + ], + "allcaps": [ + "ado", + "all", + "c", + "dr", + "eg", + "gi", + "has", + "it", + "la", + "mit", + "no", + "not", + "now", + "to", + "un", + "us", + "usd" + ], + "abbrev": [ + "aaa", + "abc", + "ac", + "acsi", + "adf", + "adhd", + "ado", + "adsl", + "ae", + "aes", + "afk", + "agm", + "agpl", + "ai", + "aitb", + "aj", + "amd", + "aol", + "aph", + "api", + "asap", + "aspca", + "astm", + "ati", + "atk", + "atm", + "atsc", + "atv", + "avc", + "avg", + "bsod", + "byod", + "c", + "ceo", + "ces", + "cet", + "cia", + "cio", + "ctia", + "diy", + "dmca", + "doj", + "dr", + "echr", + "edst", + "edt", + "eff", + "efi", + "eg", + "ept", + "espn", + "esrb", + "est", + "eu", + "evga", + "ewmh", + "exe", + "ftaa", + "fyi", + "gi", + "hiv", + "ibm", + "ibmtts", + "icmp", + "ieee", + "ietf", + "iidc", + "iis", + "iiuc", + "imo", + "inlb", + "ios", + "ip", + "ipa", + "ipcc", + "iptv", + "irc", + "irs", + "isbn", + "isp", + "it", + "itx", + "la", + "lapd", + "lbs", + "lotr", + "mit", + "mmorpg", + "motd", + "mpaa", + "msaa", + "mya", + "ncis", + "ntia", + "ny", + "nypd", + "nypsd", + "nyse", + "nyt", + "ocd", + "ocr", + "odf", + "oecd", + "oic", + "ok", + "olpc", + "omg", + "os", + "osx", + "otoh", + "otr", + "pcie", + "pcmcia", + "psa", + "psu", + "riaa", + "rnib", + "rtos", + "sae", + "sla", + "ssid", + "ssip", + "suv", + "ucl", + "ucla", + "ucs", + "udp", + "ueb", + "uefi", + "ufo", + "ui", + "uia", + "uk", + "umts", + "un", + "unhcr", + "upc", + "upnp", + "url", + "us", + "usa", + "usaf", + "usb", + "usd", + "usda", + "ussr", + "utc", + "utf", + "uucp", + "uuid", + "uv", + "vi", + "wuxga", + "xl", + "xxx", + "xy", + "yd" + ] + }, + "letters": { + "a": "ˈeɪ", + "b": "bˈiː", + "c": "sˈiː", + "d": "dˈiː", + "e": "ˈiː", + "f": "ˈɛf", + "g": "dʒˈiː", + "h": "ˈeɪtʃ", + "i": "ˈaɪ", + "j": "dʒˈeɪ", + "k": "kˈeɪ", + "l": "ˈɛl", + "m": "ˈɛm", + "n": "ˈɛn", + "o": "ˈoʊ", + "p": "pˈiː", + "q": "kjˈuː", + "r": "ˈɑːɹ", + "s": "ˈɛs", + "t": "tˈiː", + "u": "jˈuː", + "v": "vˈiː", + "w": "dˈʌbəljˌuː", + "x": "ˈɛks", + "y": "wˈaɪ", + "z": "zˈiː" + } +} \ No newline at end of file diff --git a/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_lexicon.tsv.zz b/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_lexicon.tsv.zz new file mode 100644 index 00000000..6ebc4f54 Binary files /dev/null and b/Sources/FluidAudio/TTS/LuxTts/G2p/Resources/luxtts_en_us_lexicon.tsv.zz differ diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift new file mode 100644 index 00000000..6a7aa666 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift @@ -0,0 +1,57 @@ +import Foundation + +/// Compile-time constants for the LuxTTS (ZipVoice-Distill) backend. +/// +/// Values mirror the upstream LuxTTS inference defaults and the fixed-shape +/// buckets baked into the CoreML graphs at +/// `FluidInference/luxtts-coreml` (see the repo README for the bucket table). +public enum LuxTtsConstants { + + /// Sample rate of the mel frontend / prompt conditioning (Hz). + public static let melSampleRate = 24000 + /// Sample rate of the generated waveform (Hz) — the vocoder upsamples + /// 24 kHz mel frames to 48 kHz audio in-graph. + public static let outputSampleRate = 48000 + + // Mel frontend (upstream VocosFbank: torchaudio MelSpectrogram, power=1). + public static let nFFT = 1024 + public static let hopLength = 256 + public static let nMels = 100 + /// Log floor: `mel.clamp(min: 1e-7).log()`. + public static let logMelFloor: Float = 1e-7 + /// Features are scaled by 0.1 before conditioning (`feat_scale`). + public static let featScale: Float = 0.1 + + // Fixed CoreML shape buckets (gpu/ + ane/ graphs). + public static let maxTokens = 256 + public static let maxFrames = 1024 + public static let featDim = 100 + + // Flow-matching solver. + public static let numSteps = 4 + public static let tShift = 0.5 + public static let guidanceScale: Float = 3.0 + + /// Default speech-rate divisor. Upstream `generate()` silently multiplies + /// speed by 1.3, which squeezes the ratio-based duration estimate and + /// clips sentence onsets; 1.0 synthesizes complete sentences. + public static let defaultSpeed: Float = 1.0 + + /// Prompt RMS normalization target (upstream `rms_norm`). Prompts quieter + /// than this are boosted before mel extraction and the generated waveform + /// is scaled back down by the same factor. + public static let targetRms: Float = 0.1 + + /// Prompt duration cap in seconds. Frames beyond this would eat too much + /// of the 1024-frame bucket (~10.9 s total at 93.75 frames/s). + public static let maxPromptSeconds: Double = 5.0 + + /// Published fixed-shape vocoder buckets (generated frames). + public static let vocoderBuckets = [282, 555] + /// Vocoder hop at 48 kHz (256 at 24 kHz × 2). The vocoder emits + /// `(bucket - 1) * hop48k` samples. + public static let hop48k = 512 + + /// Default synthesis noise seed (matches the Python reference scripts). + public static let defaultSeed: UInt64 = 42 +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsError.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsError.swift new file mode 100644 index 00000000..92782e29 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsError.swift @@ -0,0 +1,40 @@ +import Foundation + +/// Errors surfaced by the LuxTTS backend. +public enum LuxTtsError: Error, LocalizedError { + case notInitialized + case downloadFailed(String) + case modelFileNotFound(String) + case corruptedModel(String, underlying: String) + case tokenizerFailed(String) + case invalidPromptAudio(String) + case inputTooLong(String) + case degenerateDuration(featuresLength: Int, tokensCount: Int) + case inferenceFailed(stage: String, underlying: String) + + public var errorDescription: String? { + switch self { + case .notInitialized: + return "LuxTTS is not initialized. Call initialize() first." + case .downloadFailed(let detail): + return "LuxTTS model download failed: \(detail)" + case .modelFileNotFound(let name): + return "LuxTTS model file not found: \(name)" + case .corruptedModel(let name, let underlying): + return "LuxTTS model \(name) failed to load: \(underlying)" + case .tokenizerFailed(let detail): + return "LuxTTS tokenizer failure: \(detail)" + case .invalidPromptAudio(let detail): + return "LuxTTS prompt audio invalid: \(detail)" + case .inputTooLong(let detail): + return "LuxTTS input exceeds the fixed CoreML shape bucket: \(detail)" + case .degenerateDuration(let featuresLength, let tokensCount): + return + "LuxTTS duration estimate is degenerate: features length \(featuresLength) < " + + "token count \(tokensCount) yields < 1 frame per token, which would collapse " + + "every frame onto the pad slot (prompt too short or too many tokens)" + case .inferenceFailed(let stage, let underlying): + return "LuxTTS inference failed at \(stage): \(underlying)" + } + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift new file mode 100644 index 00000000..786d53bc --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift @@ -0,0 +1,189 @@ +@preconcurrency import CoreML +import Foundation + +/// Top-level public API for LuxTTS (ZipVoice-Distill) zero-shot +/// voice-cloning TTS — 48 kHz output conditioned on a short prompt clip. +/// +/// Pipeline pieces: +/// 1. `LuxTtsModelStore` — downloads + holds the CoreML stages +/// (TextEncoder, FmDecoder, fixed-shape Vocos vocoders) and `tokens.txt`. +/// 2. `LuxTtsTokenizer` — espeak-IPA phoneme string → token ids. +/// 3. `LuxTtsSynthesizer` — flow-matching host loop (see its docs). +/// +/// Text input runs through `LuxTtsG2p` (espeak-parity English G2P from a +/// bundled lexicon — the model was trained on espeak `en-us` phonemes via +/// EmiliaTokenizer, and Misaki-style frontends do not map onto that token +/// set). Pre-phonemized espeak IPA is still accepted via +/// `synthesize(phonemes:...)`. +/// +/// Usage: +/// ```swift +/// let manager = try await LuxTtsManager.downloadAndCreate() +/// let result = try await manager.synthesize( +/// text: "The quick brown fox jumps over the lazy dog.", +/// promptAudio: promptWavURL, +/// promptText: "The transcript of the prompt clip.") +/// // result.samples is 48 kHz mono Float32 PCM. +/// ``` +public actor LuxTtsManager { + + private let logger = AppLogger(category: "LuxTtsManager") + + private let directory: URL? + private let variant: String + private let computeUnitsOverride: MLComputeUnits? + + private var store: LuxTtsModelStore? + private var synthesizer: LuxTtsSynthesizer? + private var g2p: LuxTtsG2p? + + /// - Parameters: + /// - directory: Model cache root override (default: shared TTS cache). + /// - variant: Graph variant (`ModelNames.LuxTts.gpuVariant` / + /// `.aneVariant`). Defaults to the platform-appropriate graph: + /// `gpu/` + `.cpuAndGPU` on macOS, `ane/` + `.cpuAndNeuralEngine` + /// elsewhere. The `gpu/` graph must never run on the ANE (rel-pos + /// attention corrupts audio there). + /// - computeUnitsOverride: Force specific compute units for every stage. + public init( + directory: URL? = nil, + variant: String = ModelNames.LuxTts.defaultVariant, + computeUnitsOverride: MLComputeUnits? = nil + ) { + self.directory = directory + self.variant = variant + self.computeUnitsOverride = computeUnitsOverride + } + + public var isAvailable: Bool { synthesizer != nil } + + /// Convenience factory: download assets and return a ready-to-use manager. + public static func downloadAndCreate( + cacheDirectory: URL? = nil, + variant: String = ModelNames.LuxTts.defaultVariant, + computeUnitsOverride: MLComputeUnits? = nil + ) async throws -> LuxTtsManager { + let manager = LuxTtsManager( + directory: cacheDirectory, + variant: variant, + computeUnitsOverride: computeUnitsOverride) + try await manager.initialize() + return manager + } + + /// Download (if missing) and load the LuxTTS CoreML stages. + public func initialize(progressHandler: ProgressHandler? = nil) async throws { + if synthesizer != nil { return } + + let store = LuxTtsModelStore( + directory: directory, + variant: variant, + computeUnitsOverride: computeUnitsOverride) + try await store.loadIfNeeded(progressHandler: progressHandler) + + self.store = store + self.synthesizer = LuxTtsSynthesizer(store: store) + logger.info("LuxTTS ready (variant: \(variant))") + } + + // MARK: - Synthesis + + /// Synthesize from raw English text (espeak-parity G2P, see `LuxTtsG2p`). + /// + /// - Parameters: + /// - text: English text to speak. + /// - promptAudio: Prompt clip (see `synthesize(phonemes:...)`). + /// - promptText: Transcript of the prompt clip (raw text). + public func synthesize( + text: String, + promptAudio: URL, + promptText: String, + speed: Float = LuxTtsConstants.defaultSpeed, + seed: UInt64 = LuxTtsConstants.defaultSeed + ) async throws -> LuxTtsSynthesisResult { + // Fail fast before the (potentially expensive) G2P lexicon load and + // phonemization; the phonemes path guards on the same store below. + guard store != nil else { throw LuxTtsError.notInitialized } + let g2p = try englishG2p() + return try await synthesize( + phonemes: g2p.phonemize(text: text), + promptAudio: promptAudio, + promptPhonemes: g2p.phonemize(text: promptText), + speed: speed, + seed: seed) + } + + /// The bundled espeak-parity English G2P (loaded lazily; ~4 MB of + /// lexicon tables, no network access). + public func englishG2p() throws -> LuxTtsG2p { + if let g2p { return g2p } + let g2p = try LuxTtsG2p() + self.g2p = g2p + return g2p + } + + /// Synthesize from espeak-IPA phoneme strings (the `tokens.txt` set; + /// one token per Unicode scalar, OOV scalars skipped with a warning). + /// + /// - Parameters: + /// - phonemes: espeak IPA for the text to speak. + /// - promptAudio: Prompt clip (any format/rate; converted to 24 kHz + /// mono, capped at `LuxTtsConstants.maxPromptSeconds`). Trim + /// leading/trailing silence beforehand (e.g. with `VadManager`) — + /// silence inflates the frames-per-token duration ratio. + /// - promptPhonemes: espeak IPA of the prompt clip's transcript. + /// - speed: Speech-rate divisor for the generated span. Keep 1.0 + /// (upstream's hidden 1.3 clips sentence onsets). + /// - seed: Noise seed for the flow-matching init. + public func synthesize( + phonemes: String, + promptAudio: URL, + promptPhonemes: String, + speed: Float = LuxTtsConstants.defaultSpeed, + seed: UInt64 = LuxTtsConstants.defaultSeed + ) async throws -> LuxTtsSynthesisResult { + guard let store = store else { throw LuxTtsError.notInitialized } + let tokenizer = try await store.tokenizer() + return try await synthesize( + tokenIds: tokenizer.tokenIds(phonemes: phonemes), + promptAudio: promptAudio, + promptTokenIds: tokenizer.tokenIds(phonemes: promptPhonemes), + speed: speed, + seed: seed) + } + + /// Synthesize from pre-computed token ids (callers running their own + /// espeak frontend against `tokens.txt`). + public func synthesize( + tokenIds: [Int], + promptAudio: URL, + promptTokenIds: [Int], + speed: Float = LuxTtsConstants.defaultSpeed, + seed: UInt64 = LuxTtsConstants.defaultSeed + ) async throws -> LuxTtsSynthesisResult { + guard let synthesizer = synthesizer else { throw LuxTtsError.notInitialized } + + let prompt24k: [Float] + do { + let converter = AudioConverter( + sampleRate: Double(LuxTtsConstants.melSampleRate)) + prompt24k = try converter.resampleAudioFile(promptAudio) + } catch { + throw LuxTtsError.invalidPromptAudio( + "cannot load \(promptAudio.path): \(error.localizedDescription)") + } + + return try await synthesizer.synthesize( + promptTokenIds: promptTokenIds, + textTokenIds: tokenIds, + promptAudio24k: prompt24k, + speed: speed, + seed: seed) + } + + public func cleanup() async { + if let store = store { await store.unload() } + store = nil + synthesizer = nil + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsMelExtractor.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsMelExtractor.swift new file mode 100644 index 00000000..49db407a --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsMelExtractor.swift @@ -0,0 +1,188 @@ +import Accelerate +import Foundation + +/// Native Swift port of the upstream LuxTTS mel frontend (`VocosFbank`): +/// `torchaudio.transforms.MelSpectrogram(sample_rate: 24000, n_fft: 1024, +/// hop_length: 256, n_mels: 100, center: true, power: 1)` followed by +/// `log(clamp(min: 1e-7))`, with the frame count adjusted to lhotse's +/// `compute_num_frames` (truncate, or replicate-pad the last frame). +/// +/// torchaudio specifics mirrored here (they differ from the NeMo-flavoured +/// `AudioMelSpectrogram`): periodic Hann window, reflect padding, magnitude +/// (power = 1) spectrum, HTK mel scale with no filterbank normalization. +/// +/// Structure mirrors `NemotronMelExtractor` / `AudioMelSpectrogram`. +public final class LuxTtsMelExtractor { + + private let nFFT = LuxTtsConstants.nFFT + private let hop = LuxTtsConstants.hopLength + private let nMels = LuxTtsConstants.nMels + private let sampleRate = LuxTtsConstants.melSampleRate + + private let hannWindow: [Float] + private let melFilterbankFlat: [Float] // [nMels x (nFFT/2+1)] row-major + // Immutable after init (only read in extract, freed in deinit). + private let fftSetup: vDSP_DFT_Setup? + + public init() { + // The window/filterbank are call-independent pure-value tables; pull + // them from a process-wide Sendable cache so repeated construction + // (the synthesizer builds one extractor per call to stay Sendable) + // only pays for the cheap vDSP DFT setup, not the table math. + let tables = Self.sharedTables + self.hannWindow = tables.hannWindow + self.melFilterbankFlat = tables.melFilterbankFlat + self.fftSetup = vDSP_DFT_zop_CreateSetup(nil, vDSP_Length(nFFT), .FORWARD) + } + + deinit { + if let setup = fftSetup { + vDSP_DFT_DestroySetup(setup) + } + } + + /// lhotse `compute_num_frames`: `(num_samples + hop/2) / hop`. + public func frameCount(sampleCount: Int) -> Int { + (sampleCount + hop / 2) / hop + } + + /// Compute the log-mel spectrogram of 24 kHz mono audio. + /// - Returns: `[T][nMels]` log-mel frames (unscaled — the caller applies + /// `LuxTtsConstants.featScale`), `T = frameCount(sampleCount:)`. + public func extract(audio: [Float]) -> [[Float]] { + let n = audio.count + let targetFrames = frameCount(sampleCount: n) + guard n > 0, targetFrames > 0, let setup = fftSetup else { return [] } + + // Reflect-pad by nFFT/2 on both sides (torch pad_mode="reflect"). + let pad = nFFT / 2 + var padded = [Float](repeating: 0, count: n + 2 * pad) + for i in 0.. [Float] { + (0.. [Float] { + let bins = nFFT / 2 + 1 + let fMax = Double(sampleRate) / 2.0 + + func hzToMel(_ hz: Double) -> Double { 2595.0 * log10(1.0 + hz / 700.0) } + func melToHz(_ mel: Double) -> Double { 700.0 * (pow(10.0, mel / 2595.0) - 1.0) } + + let melMin = hzToMel(0) + let melMax = hzToMel(fMax) + let melPoints = (0..<(nMels + 2)).map { i in + melToHz(melMin + Double(i) * (melMax - melMin) / Double(nMels + 1)) + } + // all_freqs = linspace(0, sr/2, bins) + let freqs = (0.. URL { + let modelsRoot = try directory ?? defaultCacheRoot() + let repoDir = modelsRoot.appendingPathComponent(Repo.luxtts.folderName) + + let required = ModelNames.LuxTts.requiredFiles(variant: variant) + let allPresent = required.allSatisfy { file in + FileManager.default.fileExists(atPath: repoDir.appendingPathComponent(file).path) + } + + if !allPresent { + logger.info("Downloading LuxTTS CoreML assets (\(variant)/) from HuggingFace…") + do { + try await ModelHub.download( + .luxtts, to: modelsRoot, variant: variant, + progressHandler: progressHandler) + } catch { + throw LuxTtsError.downloadFailed("\(error)") + } + } else { + logger.info("LuxTTS assets found in cache at \(repoDir.path)") + } + + return repoDir + } + + private static func defaultCacheRoot() throws -> URL { + let root = try TtsCacheDirectory.ensure().appendingPathComponent("Models") + if !FileManager.default.fileExists(atPath: root.path) { + try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true) + } + return root + } +} + +/// Actor-based store for the LuxTTS CoreML models: text encoder + +/// flow-matching decoder (per-platform graph variant) and the two +/// fixed-shape vocoders (loaded lazily by generated-frame budget). +/// +/// Compute-unit placement is deliberate: +/// - macOS loads the `gpu/` graphs with `.cpuAndGPU`. The original graph +/// must NOT run on the ANE (rel-pos attention loses precision there and +/// corrupts audio). +/// - iOS loads the `ane/` graphs with `.cpuAndNeuralEngine` (FmDecoder +/// 100% ANE-resident, TextEncoder 99%; ~25 MB jetsam-visible footprint). +/// - The vocoder runs with `.cpuAndGPU` everywhere (ISTFT + resample + +/// crossover in-graph, ~2.3 ms on GPU). It is only ~72% ANE-placeable +/// and CPU_AND_NE compilation is flaky, so it never goes on the ANE. +/// +/// The `ane/` and `gpu/` graphs share an IDENTICAL external I/O contract — +/// same input/output names, shapes and fp32 dtype (TextEncoder: +/// `tokens (1,256) i32` + `padding_mask (1,256)` → `token_embeds (1,256,100)`; +/// FmDecoder: `x`/`text_condition`/`speech_condition (1,1024,100)` + +/// `t`/`guidance_scale (1)` + `padding_mask (1,1024)` → `v (1,1024,100)`). +/// The "ANE-canonical" rewrite (channel-axis pre-concat, `(1,C,1,S)` working +/// layout) lives entirely INSIDE the MIL and transposes back before output, +/// so `LuxTtsSynthesizer` drives both variants with the same tensor packing — +/// no host-side adapter is required. Select the variant via `variant` (and +/// the `FLUIDAUDIO_LUXTTS_VARIANT` override folded into `defaultVariant`). +public actor LuxTtsModelStore { + + private let logger = AppLogger(category: "LuxTtsModelStore") + + private let directory: URL? + private let computeUnitsOverride: MLComputeUnits? + let variant: String + + private var repoDirectory: URL? + private var textEncoderModel: MLModel? + private var fmDecoderModel: MLModel? + private var vocoderModels: [Int: MLModel] = [:] + private var loadedTokenizer: LuxTtsTokenizer? + + public init( + directory: URL? = nil, + variant: String = ModelNames.LuxTts.defaultVariant, + computeUnitsOverride: MLComputeUnits? = nil + ) { + self.directory = directory + self.variant = variant + self.computeUnitsOverride = computeUnitsOverride + } + + private var encoderDecoderComputeUnits: MLComputeUnits { + Self.encoderDecoderComputeUnits(variant: variant, override: computeUnitsOverride) + } + + private var vocoderComputeUnits: MLComputeUnits { + computeUnitsOverride ?? .cpuAndGPU + } + + /// Encoder/decoder compute-unit policy (pure; unit-tested). `ane/` → ANE, + /// everything else → GPU. An explicit `computeUnitsOverride` wins. + static func encoderDecoderComputeUnits( + variant: String, override: MLComputeUnits? + ) -> MLComputeUnits { + if let override { return override } + return variant == ModelNames.LuxTts.aneVariant ? .cpuAndNeuralEngine : .cpuAndGPU + } + + // MARK: - Loading + + public func loadIfNeeded(progressHandler: ProgressHandler? = nil) async throws { + if textEncoderModel != nil { return } + + let repoDir = try await LuxTtsResourceDownloader.ensureModels( + directory: directory, variant: variant, progressHandler: progressHandler) + self.repoDirectory = repoDir + + logger.info("Loading LuxTTS CoreML models (\(variant)/) from \(repoDir.path)…") + let loadStart = Date() + + let config = MLModelConfiguration() + config.computeUnits = encoderDecoderComputeUnits + + textEncoderModel = try loadModel( + repoDir: repoDir, + fileName: ModelNames.LuxTts.textEncoderFile(variant: variant), config: config) + fmDecoderModel = try loadModel( + repoDir: repoDir, + fileName: ModelNames.LuxTts.fmDecoderFile(variant: variant), config: config) + + loadedTokenizer = try LuxTtsTokenizer( + tokensFileURL: repoDir.appendingPathComponent(ModelNames.LuxTts.tokensFile)) + + let elapsed = Date().timeIntervalSince(loadStart) + logger.info("LuxTTS models loaded in \(String(format: "%.2f", elapsed))s") + } + + // MARK: - Accessors + + public func textEncoder() throws -> MLModel { + guard let model = textEncoderModel else { throw LuxTtsError.notInitialized } + return model + } + + public func fmDecoder() throws -> MLModel { + guard let model = fmDecoderModel else { throw LuxTtsError.notInitialized } + return model + } + + public func tokenizer() throws -> LuxTtsTokenizer { + guard let tokenizer = loadedTokenizer else { throw LuxTtsError.notInitialized } + return tokenizer + } + + /// Vocoder for a window of `bucket` generated frames (282 or 555). + /// Loaded lazily and cached — short utterances never pay for the 555 + /// bucket, long ones never pay for the 282 one. + public func vocoder(bucket: Int) throws -> MLModel { + if let cached = vocoderModels[bucket] { return cached } + guard let repoDir = repoDirectory else { throw LuxTtsError.notInitialized } + let fileName = + bucket == 282 + ? ModelNames.LuxTts.vocoder282File : ModelNames.LuxTts.vocoder555File + let config = MLModelConfiguration() + config.computeUnits = vocoderComputeUnits + let model = try loadModel(repoDir: repoDir, fileName: fileName, config: config) + vocoderModels[bucket] = model + return model + } + + public func unload() { + textEncoderModel = nil + fmDecoderModel = nil + vocoderModels.removeAll() + loadedTokenizer = nil + } + + // MARK: - Helpers + + private func loadModel( + repoDir: URL, fileName: String, config: MLModelConfiguration + ) throws -> MLModel { + let modelURL = repoDir.appendingPathComponent(fileName) + guard FileManager.default.fileExists(atPath: modelURL.path) else { + throw LuxTtsError.modelFileNotFound(fileName) + } + do { + let model = try MLModel(contentsOf: modelURL, configuration: config) + logger.info("Loaded \(fileName)") + return model + } catch { + throw LuxTtsError.corruptedModel(fileName, underlying: "\(error)") + } + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSolver.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSolver.swift new file mode 100644 index 00000000..0bf00909 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSolver.swift @@ -0,0 +1,66 @@ +import Foundation + +/// Pure host-side math for the LuxTTS pipeline: ratio-based duration +/// estimation, average-token-duration expansion indices, anchor-Euler +/// solver timesteps and state update. All functions mirror the upstream +/// ZipVoice implementations exactly (fixture-pinned). +public enum LuxTtsSolver { + + /// Total feature length: prompt frames + ratio-estimated generated frames. + /// Mirrors `prompt_len + ceil(prompt_len / prompt_tokens * text_tokens / speed)`. + public static func featuresLength( + promptFrames: Int, promptTokenCount: Int, textTokenCount: Int, speed: Double + ) -> Int { + let generated = + Double(promptFrames) / Double(promptTokenCount) * Double(textTokenCount) / speed + return promptFrames + Int(generated.rounded(.up)) + } + + /// Solver timesteps: `t_shift * u / (1 + (t_shift - 1) * u)` over + /// `linspace(0, 1, numSteps + 1)` (upstream `get_time_steps`). + public static func timeSteps(numSteps: Int, tShift: Double) -> [Double] { + (0...numSteps).map { i in + let u = Double(i) / Double(numSteps) + return tShift * u / (1.0 + (tShift - 1.0) * u) + } + } + + /// Per-frame token index for the duration expansion. Mirrors upstream + /// `prepare_avg_tokens_durations` + `get_tokens_index`: every token gets + /// `featuresLength / tokensCount` frames and the remainder frames map to + /// index `tokensCount` — the appended pad-slot embedding row. + public static func tokensIndex(tokensCount: Int, featuresLength: Int) throws -> [Int] { + let avg = featuresLength / tokensCount + // avg < 1 collapses every real token to zero frames: the whole + // sequence maps to the pad slot and synthesis is silent. Fail loudly + // instead of emitting garbage. + guard avg >= 1 else { + throw LuxTtsError.degenerateDuration( + featuresLength: featuresLength, tokensCount: tokensCount) + } + var index = [Int](repeating: tokensCount, count: featuresLength) + var frame = 0 + for token in 0.. [Double] { + precondition(x.count == v.count) + return zip(x, v).map { xi, vi in + let x1p = xi + (1.0 - tCur) * vi + if isLast { return x1p } + let x0p = xi - tCur * vi + return (1.0 - tNext) * x0p + tNext * x1p + } + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift new file mode 100644 index 00000000..9736d374 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift @@ -0,0 +1,367 @@ +import Accelerate +@preconcurrency import CoreML +import Foundation + +/// One LuxTTS synthesis run: 48 kHz mono samples plus the frame accounting +/// used by callers to report durations. +public struct LuxTtsSynthesisResult: Sendable { + /// Generated waveform, 48 kHz mono Float32 in [-1, 1]. + public let samples: [Float] + /// Always `LuxTtsConstants.outputSampleRate` (48 000). + public let sampleRate: Int + /// Prompt conditioning length in 24 kHz mel frames. + public let promptFrames: Int + /// Generated mel frames (`featuresLength - promptFrames`). + public let generatedFrames: Int + /// Total flow-matching sequence length (prompt + generated frames). + public let featuresLength: Int +} + +/// Drives the LuxTTS (ZipVoice-Distill) CoreML stages end-to-end, mirroring +/// the Python reference host (`coreml/dump_swift_fixtures.py::synth_coreml`): +/// +/// 1. `TextEncoder(tokens, padding_mask) → token_embeds`; keep `S + 1` +/// rows (the extra pad-slot row absorbs remainder frames). +/// 2. Ratio duration estimate → `featuresLength`, avg-duration expansion +/// of token embeddings to per-frame `text_condition`. +/// 3. `speech_condition` = prompt mel (feat-scaled) zero-padded to +/// `featuresLength`; frame `padding_mask` = 1.0 beyond `featuresLength`. +/// 4. 4-step anchor-Euler flow matching over seeded N(0, 1) noise via +/// `FmDecoder` (distill guidance_scale baked input, 3.0). +/// 5. Generated mel `/ featScale` → fixed-shape Vocos vocoder (282 / 555 +/// frame bucket) → 48 kHz waveform, truncated to `(gen - 1) * 512` +/// samples and clipped to [-1, 1]. +/// 6. If the prompt RMS was below `targetRms`, scale the waveform back +/// down by `promptRms / targetRms` (upstream `rms_norm` contract). +struct LuxTtsSynthesizer { + + private let logger = AppLogger(category: "LuxTtsSynthesizer") + private let store: LuxTtsModelStore + + init(store: LuxTtsModelStore) { + self.store = store + } + + /// Synthesize from pre-tokenized ids and 24 kHz mono prompt audio. + func synthesize( + promptTokenIds: [Int], + textTokenIds: [Int], + promptAudio24k: [Float], + speed: Float, + seed: UInt64 + ) async throws -> LuxTtsSynthesisResult { + guard !promptTokenIds.isEmpty else { + throw LuxTtsError.tokenizerFailed("prompt produced no tokens") + } + guard !textTokenIds.isEmpty else { + throw LuxTtsError.tokenizerFailed("text produced no tokens") + } + guard !promptAudio24k.isEmpty else { + throw LuxTtsError.invalidPromptAudio("no samples") + } + guard speed > 0 else { + throw LuxTtsError.invalidPromptAudio("speed must be > 0 (got \(speed))") + } + + let featDim = LuxTtsConstants.featDim + let maxTokens = LuxTtsConstants.maxTokens + let maxFrames = LuxTtsConstants.maxFrames + + // --- Prompt: cap length, rms-norm, mel --- + let maxPromptSamples = Int( + LuxTtsConstants.maxPromptSeconds * Double(LuxTtsConstants.melSampleRate)) + var prompt = Array(promptAudio24k.prefix(maxPromptSamples)) + + var meanSquare: Float = 0 + vDSP_measqv(prompt, 1, &meanSquare, vDSP_Length(prompt.count)) + let promptRms = sqrt(meanSquare) + guard promptRms > 0 else { + throw LuxTtsError.invalidPromptAudio("prompt audio is silent") + } + if promptRms < LuxTtsConstants.targetRms { + var gain = LuxTtsConstants.targetRms / promptRms + vDSP_vsmul(prompt, 1, &gain, &prompt, 1, vDSP_Length(prompt.count)) + } + + // The extractor holds a non-Sendable vDSP DFT setup; keeping it a + // per-call local (rather than a stored property) keeps the whole + // synthesizer `struct` `Sendable`, so this async method can be called + // across `LuxTtsManager`'s actor boundary without tripping the + // region-based isolation / sending checker. Its precomputed tables are + // pulled from a shared cache, so construction is cheap. + let extractor = LuxTtsMelExtractor() + let promptMel = extractor.extract(audio: prompt) // [T][100], unscaled + let promptFrames = promptMel.count + guard promptFrames > 0 else { + throw LuxTtsError.invalidPromptAudio("prompt too short for one mel frame") + } + + // --- Duration estimate + shape-bucket guards --- + let cat = promptTokenIds + textTokenIds + let tokenCount = cat.count + guard tokenCount + 1 <= maxTokens else { + throw LuxTtsError.inputTooLong( + "prompt + text tokens \(tokenCount) + pad slot > \(maxTokens)") + } + + let featuresLength = LuxTtsSolver.featuresLength( + promptFrames: promptFrames, + promptTokenCount: promptTokenIds.count, + textTokenCount: textTokenIds.count, + speed: Double(speed)) + guard featuresLength <= maxFrames else { + throw LuxTtsError.inputTooLong( + "features length \(featuresLength) > \(maxFrames) frames " + + "(shorter text or shorter prompt required)") + } + let genFrames = featuresLength - promptFrames + guard genFrames >= 2 else { + throw LuxTtsError.tokenizerFailed( + "estimated only \(genFrames) generated frames — text too short") + } + // TODO(phase 2): chunk long inputs across multiple vocoder windows + // instead of erroring; mel truncation is NOT allowed. + guard let bucket = LuxTtsConstants.vocoderBuckets.first(where: { $0 >= genFrames }) else { + throw LuxTtsError.inputTooLong( + "generated frames \(genFrames) exceed the largest vocoder bucket " + + "(\(LuxTtsConstants.vocoderBuckets.max() ?? 0))") + } + + // --- Stage 1: text encoder --- + let tokens = try makeArray(shape: [1, maxTokens], dataType: .int32) + let tokenMask = try makeArray(shape: [1, maxTokens], dataType: .float32) + tokens.withUnsafeMutableBufferPointer(ofType: Int32.self) { buf, _ in + for i in 0.. [Float] { + var out = [Float](repeating: 0, count: rowCount * rowLength) + try copyRows(from: array, into: &out, rowCount: rowCount, rowLength: rowLength, stage: stage) + return out + } + + /// `copyRows` variant that writes into a caller-owned buffer (reused + /// across the flow-matching steps to avoid a per-step allocation). `out` + /// must hold at least `rowCount * rowLength` floats. + private func copyRows( + from array: MLMultiArray, into out: inout [Float], + rowCount: Int, rowLength: Int, stage: String + ) throws { + precondition(out.count >= rowCount * rowLength, "copyRows destination too small") + let dims = array.shape.count + guard array.dataType == .float32, dims >= 2, + array.strides[dims - 1].intValue == 1, + rowLength <= array.shape[dims - 1].intValue, + rowCount <= array.shape[dims - 2].intValue + else { + throw LuxTtsError.inferenceFailed( + stage: stage, + underlying: + "unexpected output layout: shape \(array.shape) strides \(array.strides) " + + "dtype \(array.dataType.rawValue) for \(rowCount)×\(rowLength) read") + } + let rowStride = array.strides[dims - 2].intValue + array.withUnsafeBufferPointer(ofType: Float.self) { src in + out.withUnsafeMutableBufferPointer { dst in + for row in 0.. MLMultiArray { + do { + let array = try MLMultiArray(shape: shape.map { NSNumber(value: $0) }, dataType: dataType) + array.reset(to: 0) + return array + } catch { + throw LuxTtsError.inferenceFailed(stage: "allocation", underlying: "\(error)") + } + } + + private func predict( + stage: String, model: MLModel, inputs: [String: MLFeatureValue] + ) throws -> MLFeatureProvider { + do { + let provider = try MLDictionaryFeatureProvider(dictionary: inputs) + return try model.prediction(from: provider) + } catch { + throw LuxTtsError.inferenceFailed(stage: stage, underlying: "\(error)") + } + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsTokenizer.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsTokenizer.swift new file mode 100644 index 00000000..6e94293d --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsTokenizer.swift @@ -0,0 +1,74 @@ +import Foundation + +/// Phoneme-token table for LuxTTS (`tokens.txt`, EmiliaTokenizer format: +/// one `{token}\t{id}` pair per line). +/// +/// The English token set is per-character espeak IPA (stress marks, length +/// marks and combining diacritics are separate single-scalar tokens), so a +/// raw espeak phoneme string maps to ids scalar-by-scalar. Multi-character +/// entries (the pinyin initial/final tokens for Mandarin) are only reachable +/// through `tokenIds(tokens:)`. +public struct LuxTtsTokenizer: Sendable { + + private static let logger = AppLogger(category: "LuxTtsTokenizer") + + public let tokenToId: [String: Int] + public let padId: Int + public var vocabSize: Int { tokenToId.count } + + public init(tokensFileURL: URL) throws { + let content: String + do { + content = try String(contentsOf: tokensFileURL, encoding: .utf8) + } catch { + throw LuxTtsError.tokenizerFailed( + "cannot read \(tokensFileURL.path): \(error.localizedDescription)") + } + + var map: [String: Int] = [:] + for line in content.split(separator: "\n", omittingEmptySubsequences: true) { + // Split on the FIRST tab only: the space token line is a literal + // space character followed by "\t3". + guard let tab = line.firstIndex(of: "\t"), let id = Int(line[line.index(after: tab)...]) + else { + throw LuxTtsError.tokenizerFailed("malformed tokens.txt line: \(line)") + } + let token = String(line[.. [Int] { + var ids: [Int] = [] + ids.reserveCapacity(phonemes.unicodeScalars.count) + var skipped: Set = [] + for scalar in phonemes.unicodeScalars { + let token = String(scalar) + if let id = tokenToId[token] { + ids.append(id) + } else { + skipped.insert(token) + } + } + if !skipped.isEmpty { + Self.logger.warning("Skipped OOV phoneme tokens: \(skipped.sorted().joined(separator: " "))") + } + return ids + } + + /// Map explicit token strings (e.g. pinyin initials/finals) to ids, + /// skipping OOV entries like upstream. + public func tokenIds(tokens: [String]) -> [Int] { + tokens.compactMap { tokenToId[$0] } + } +} diff --git a/Sources/FluidAudio/TTS/TtsBackend.swift b/Sources/FluidAudio/TTS/TtsBackend.swift index ee195751..978fd768 100644 --- a/Sources/FluidAudio/TTS/TtsBackend.swift +++ b/Sources/FluidAudio/TTS/TtsBackend.swift @@ -27,4 +27,13 @@ public enum TtsBackend: Sendable { /// (31 languages), 44.1 kHz mono output. Voice styling via per-voice /// JSON (`style_ttl` / `style_dp` tensors). case supertonic3 + /// LuxTTS (ZipVoice-Distill) — zero-shot voice cloning conditioned on a + /// short prompt clip + transcript. 3-stage CoreML pipeline + /// (`TextEncoder → FmDecoder ×4 anchor-Euler steps → fixed-shape Vocos + /// vocoder`), 48 kHz mono output. + /// + /// English text input runs through the bundled espeak-parity G2P + /// (`LuxTtsG2p`); pre-phonemized espeak IPA is accepted via + /// `LuxTtsManager.synthesize(phonemes:...)`. + case luxtts } diff --git a/Sources/FluidAudioCLI/Commands/TTS/LuxTtsG2pDumpCommand.swift b/Sources/FluidAudioCLI/Commands/TTS/LuxTtsG2pDumpCommand.swift new file mode 100644 index 00000000..1b2f717e --- /dev/null +++ b/Sources/FluidAudioCLI/Commands/TTS/LuxTtsG2pDumpCommand.swift @@ -0,0 +1,82 @@ +#if os(macOS) +import FluidAudio +import Foundation + +/// Dump LuxTTS G2P output for a sentence corpus as JSONL +/// (`{"text": ..., "phonemes": ..., "ids": [...]}` per line). +/// +/// This feeds the espeak-oracle gate in +/// `mobius/models/tts/zipvoice/coreml/g2p/validate.py`: +/// +/// swift run fluidaudiocli luxtts-g2p-dump \ +/// --corpus corpus_en_1000.txt --tokens tokens.txt --out swift_dump.jsonl +/// python -m coreml.g2p.validate score \ +/// --oracle oracle_tokens.jsonl --swift swift_dump.jsonl +enum LuxTtsG2pDumpCommand { + + private static let logger = AppLogger(category: "LuxTtsG2pDump") + + static func run(arguments: [String]) async { + var corpusPath: String? = nil + var outPath: String? = nil + var tokensPath: String? = nil + var i = 0 + while i < arguments.count { + switch arguments[i] { + case "--corpus": + if i + 1 < arguments.count { + corpusPath = arguments[i + 1] + i += 1 + } + case "--out": + if i + 1 < arguments.count { + outPath = arguments[i + 1] + i += 1 + } + case "--tokens": + if i + 1 < arguments.count { + tokensPath = arguments[i + 1] + i += 1 + } + case "--help", "-h": + print( + "Usage: fluidaudiocli luxtts-g2p-dump --corpus " + + "--tokens --out ") + return + default: + break + } + i += 1 + } + guard let corpusPath, let outPath, let tokensPath else { + logger.error("luxtts-g2p-dump requires --corpus, --tokens and --out") + exit(1) + } + do { + let g2p = try LuxTtsG2p() + let tokenizer = try LuxTtsTokenizer( + tokensFileURL: URL(fileURLWithPath: tokensPath)) + + let corpus = try String(contentsOfFile: corpusPath, encoding: .utf8) + var lines: [String] = [] + for sentence in corpus.split(separator: "\n", omittingEmptySubsequences: true) { + let text = sentence.trimmingCharacters(in: .whitespaces) + if text.isEmpty { continue } + let phonemes = g2p.phonemize(text: text) + let ids = tokenizer.tokenIds(phonemes: phonemes) + let record: [String: Any] = [ + "text": text, "phonemes": phonemes, "ids": ids, + ] + let data = try JSONSerialization.data(withJSONObject: record) + lines.append(String(data: data, encoding: .utf8)!) + } + try (lines.joined(separator: "\n") + "\n") + .write(toFile: outPath, atomically: true, encoding: .utf8) + logger.info("Wrote \(lines.count) entries to \(outPath)") + } catch { + logger.error("luxtts-g2p-dump failed: \(error)") + exit(1) + } + } +} +#endif diff --git a/Sources/FluidAudioCLI/Commands/TTSCommand.swift b/Sources/FluidAudioCLI/Commands/TTSCommand.swift index 287453c9..d4819333 100644 --- a/Sources/FluidAudioCLI/Commands/TTSCommand.swift +++ b/Sources/FluidAudioCLI/Commands/TTSCommand.swift @@ -96,6 +96,11 @@ public struct TTS { // VectorEstimator build: fp16 | int8/int6/int4 (ANE-bucketed) | // dyn-int8/dyn-int6/dyn-int4 (dynamic CPU/GPU). Default fp16. var supertonicVE: Supertonic3VectorEstimator = .aneBucketed(.int4) + // LuxTTS zero-shot voice-cloning args. + var luxttsPromptAudioPath: String? = nil + var luxttsPromptText: String? = nil + var luxttsSpeed: Float = LuxTtsConstants.defaultSpeed + var luxttsSeed: UInt64 = LuxTtsConstants.defaultSeed var i = 0 while i < arguments.count { @@ -153,6 +158,8 @@ public struct TTS { backend = .styletts2 case "supertonic3", "supertonic-3", "sup3": backend = .supertonic3 + case "luxtts", "lux-tts", "lux", "zipvoice": + backend = .luxtts default: logger.warning("Unknown backend '\(arguments[i + 1])'; using kokoro-ane") } @@ -188,6 +195,17 @@ public struct TTS { case "--speed": if i + 1 < arguments.count, let v = Float(arguments[i + 1]) { supertonicSpeed = v + luxttsSpeed = v + i += 1 + } + case "--prompt-audio": + if i + 1 < arguments.count { + luxttsPromptAudioPath = arguments[i + 1] + i += 1 + } + case "--prompt-text": + if i + 1 < arguments.count { + luxttsPromptText = arguments[i + 1] i += 1 } case "--silence": @@ -219,6 +237,7 @@ public struct TTS { if i + 1 < arguments.count, let parsed = UInt64(arguments[i + 1]) { styletts2Seed = parsed pocketSeed = parsed + luxttsSeed = parsed i += 1 } case "--cpu-only": @@ -323,7 +342,178 @@ public struct TTS { silenceDuration: supertonicSilence, vectorEstimator: supertonicVE, metricsPath: metricsPath, cpuOnly: cpuOnly) + case .luxtts: + await runLuxTts( + text: text, output: output, + promptAudioPath: luxttsPromptAudioPath, + promptText: luxttsPromptText, + treatAsPhonemes: treatAsPhonemes, + speed: luxttsSpeed, seed: luxttsSeed, + metricsPath: metricsPath) + } + } + + /// Run LuxTTS zero-shot voice cloning. Requires `--prompt-audio`. + /// Text mode (default): the positional text and `--prompt-text` are + /// raw English, phonemized in-process (`LuxTtsG2p`). If `--prompt-text` + /// is omitted the prompt clip is transcribed with Parakeet ASR (models + /// download on first use). With `--phonemes`, both the text and + /// `--prompt-text` are espeak IPA (en-us) from the `tokens.txt` set. + private static func runLuxTts( + text: String, output: String, + promptAudioPath: String?, promptText: String?, + treatAsPhonemes: Bool, + speed: Float, seed: UInt64, + metricsPath: String? + ) async { + guard let promptAudioPath else { + logger.error("luxtts backend requires --prompt-audio ") + exit(1) + } + if treatAsPhonemes && promptText == nil { + logger.error( + "luxtts --phonemes requires --prompt-text (espeak IPA of the prompt " + + "clip); ASR prompt transcription is only available in text mode") + exit(1) + } + do { + let tStart = Date() + let manager = LuxTtsManager() + + let promptURL = resolveInputURL(promptAudioPath) + logger.info("LuxTTS prompt audio: \(promptURL.path)") + logger.info( + "LuxTTS speed=\(String(format: "%.2f", speed)) seed=\(seed)") + + // Prompt transcription (ASR) is independent of the TTS models and + // only needs the prompt URL, so resolve it before loading the + // synthesis stages. On failure, point the user at --prompt-text so + // they can skip ASR entirely. + let resolvedPromptText: String + if let promptText { + resolvedPromptText = promptText + } else { + do { + resolvedPromptText = try await transcribeLuxTtsPrompt(promptURL) + } catch { + logger.error( + "Prompt transcription failed; pass --prompt-text to skip ASR: \(error)") + throw error + } + } + + let tLoad0 = Date() + try await manager.initialize() + let tLoad1 = Date() + + let tSynth0 = Date() + let result: LuxTtsSynthesisResult + if treatAsPhonemes { + result = try await manager.synthesize( + phonemes: text, + promptAudio: promptURL, + promptPhonemes: resolvedPromptText, + speed: speed, + seed: seed) + } else { + result = try await manager.synthesize( + text: text, + promptAudio: promptURL, + promptText: resolvedPromptText, + speed: speed, + seed: seed) + } + let tSynth1 = Date() + + let outURL = resolveInputURL(output) + try FileManager.default.createDirectory( + at: outURL.deletingLastPathComponent(), + withIntermediateDirectories: true) + // No peak normalization: the output level carries the + // prompt-matched loudness (upstream rms_norm contract). + let wav = try AudioWAV.data( + from: result.samples, + sampleRate: Double(result.sampleRate), + normalize: false) + try wav.write(to: outURL) + + let loadS = tLoad1.timeIntervalSince(tLoad0) + let synthS = tSynth1.timeIntervalSince(tSynth0) + let totalS = tSynth1.timeIntervalSince(tStart) + let audioSecs = Double(result.samples.count) / Double(result.sampleRate) + let rtfx = synthS > 0 ? audioSecs / synthS : 0 + let sumSquares = result.samples.reduce(Double(0)) { $0 + Double($1) * Double($1) } + let rms = result.samples.isEmpty ? 0 : (sumSquares / Double(result.samples.count)).squareRoot() + + logger.info("LuxTTS synthesis complete") + logger.info(" Load: \(String(format: "%.3f", loadS))s") + logger.info(" Synthesis: \(String(format: "%.3f", synthS))s") + logger.info( + " Audio: \(String(format: "%.3f", audioSecs))s " + + "(\(result.samples.count) samples @ \(result.sampleRate) Hz)") + logger.info( + " Frames: prompt=\(result.promptFrames) " + + "generated=\(result.generatedFrames) total=\(result.featuresLength)") + logger.info(" RMS: \(String(format: "%.5f", rms))") + logger.info(" RTFx: \(String(format: "%.2f", rtfx))x") + logger.info(" Total: \(String(format: "%.3f", totalS))s") + logger.info(" Output: \(outURL.path)") + + if let metricsPath { + let metricsDict: [String: Any] = [ + "backend": "luxtts", + "text": text, + "prompt_audio": promptURL.path, + "speed": Double(speed), + "seed": seed, + "output": outURL.path, + "model_load_time_s": loadS, + "inference_time_s": synthS, + "audio_duration_s": audioSecs, + "audio_samples": result.samples.count, + "audio_rms": rms, + "prompt_frames": result.promptFrames, + "generated_frames": result.generatedFrames, + "realtime_speed": rtfx, + "total_time_s": totalS, + ] + let artifactsRoot = try ensureArtifactsRoot() + let mURL = resolveOutputURL( + metricsPath, artifactsRoot: artifactsRoot, expectsDirectory: false) + try FileManager.default.createDirectory( + at: mURL.deletingLastPathComponent(), + withIntermediateDirectories: true) + let json = try JSONSerialization.data( + withJSONObject: metricsDict, options: [.prettyPrinted]) + try json.write(to: mURL) + logger.info("Metrics saved: \(mURL.path)") + } + } catch { + logger.error("LuxTTS Error: \(error)") + print("LuxTTS failed: \(error)") + exit(1) + } + } + + /// Transcribe the LuxTTS prompt clip with Parakeet ASR (only invoked + /// when `--prompt-text` is omitted, so TTS-only users never pay the + /// ASR model download). + private static func transcribeLuxTtsPrompt(_ promptURL: URL) async throws -> String { + logger.info("--prompt-text not provided; transcribing prompt with Parakeet ASR…") + let models = try await AsrModels.downloadAndLoad() + let asrManager = AsrManager(config: .default) + try await asrManager.loadModels(models) + var decoderState = TdtDecoderState.make( + decoderLayers: await asrManager.decoderLayerCount) + let result = try await asrManager.transcribe(promptURL, decoderState: &decoderState) + let transcript = result.text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !transcript.isEmpty else { + throw LuxTtsError.invalidPromptAudio( + "ASR produced an empty transcript for \(promptURL.path); " + + "pass --prompt-text explicitly") } + logger.info("Prompt transcript: \(transcript)") + return transcript } /// Run PocketTTS in deterministic-seed mode through the session API, @@ -921,7 +1111,7 @@ public struct TTS { Options: --output, -o Output WAV path (default: output.wav) --voice, -v Voice name (default: af_heart for KokoroAne, alba for PocketTTS) - --backend TTS backend: kokoro-ane (default), pocket, styletts2, supertonic3 + --backend TTS backend: kokoro-ane (default), pocket, styletts2, supertonic3, luxtts StyleTTS2 (zero-shot, English): --reference required --alpha 0.3 ref-side blend (default 0.3) @@ -936,6 +1126,15 @@ public struct TTS { --speed 1.05 duration multiplier (default 1.05) --silence 0.05 inter-chunk silence seconds (default 0.05) --cpu-only disable Neural Engine + LuxTTS (zero-shot voice cloning, 48 kHz): + --prompt-audio required — voice prompt (<= 5 s used) + --prompt-text "…" prompt transcript (English text); if + omitted, the clip is transcribed with + Parakeet ASR (downloads ASR models) + --phonemes bypass the built-in G2P: text and + --prompt-text are espeak IPA (en-us) + --speed 1.0 speech-rate divisor (default 1.0) + --seed N flow-matching noise seed (default 42) --lexicon, -l Custom pronunciation lexicon file (KokoroAne --variant zh only): word pinyin1 pinyin2 (e.g. zi4 jie2) word @bopomofo1 (escape: @-prefixed, diff --git a/Sources/FluidAudioCLI/FluidAudioCLI.swift b/Sources/FluidAudioCLI/FluidAudioCLI.swift index 2504661e..a8bc9750 100644 --- a/Sources/FluidAudioCLI/FluidAudioCLI.swift +++ b/Sources/FluidAudioCLI/FluidAudioCLI.swift @@ -48,6 +48,8 @@ struct FluidAudioCLI { await TTS.run(arguments: Array(arguments.dropFirst(2))) case "tts-asr-verify": await TTSAsrVerifyCommand.run(arguments: Array(arguments.dropFirst(2))) + case "luxtts-g2p-dump": + await LuxTtsG2pDumpCommand.run(arguments: Array(arguments.dropFirst(2))) case "tts-benchmark": await TtsBenchmarkCommand.run(arguments: Array(arguments.dropFirst(2))) case "minimax-corpus": diff --git a/Tests/FluidAudioTests/ASR/Parakeet/ModelNamesTests.swift b/Tests/FluidAudioTests/ASR/Parakeet/ModelNamesTests.swift index 53346c41..99a62f4b 100644 --- a/Tests/FluidAudioTests/ASR/Parakeet/ModelNamesTests.swift +++ b/Tests/FluidAudioTests/ASR/Parakeet/ModelNamesTests.swift @@ -46,7 +46,9 @@ final class ModelNamesTests: XCTestCase { } func testModelFileExtensions() { - let validExtensions: Set = [".mlmodelc", ".json", ".bin"] + // `.txt` covers vocab/token text files (e.g. LuxTTS `tokens.txt`), + // which are legitimate model artifacts. + let validExtensions: Set = [".mlmodelc", ".json", ".bin", ".txt"] let validDirectories: Set = ["constants_bin"] for repo in Repo.allCases { diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift new file mode 100644 index 00000000..5464e113 --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift @@ -0,0 +1,87 @@ +import XCTest + +@testable import FluidAudio + +/// Model-dependent end-to-end synthesis test. Downloads/loads the LuxTTS +/// CoreML stages, so it only runs when explicitly requested: +/// `FLUIDAUDIO_RUN_LUXTTS_E2E=1 swift test --filter LuxTtsE2ETests`. +/// +/// Gates mirror the Python fixture run (`dump_swift_fixtures.py`): +/// identical frame accounting (same tokenizer + duration math ⇒ exact), +/// 48 kHz output, and RMS within 1 dB of the Python CoreML pipeline +/// (waveforms differ — the noise RNG is not torch's). +final class LuxTtsE2ETests: XCTestCase { + + private var shouldRunHeavy: Bool { + ProcessInfo.processInfo.environment["FLUIDAUDIO_RUN_LUXTTS_E2E"] == "1" + } + + func testSynthesizeMatchesPythonFixtureStats() async throws { + try XCTSkipUnless( + shouldRunHeavy, + "Set FLUIDAUDIO_RUN_LUXTTS_E2E=1 to run end-to-end LuxTTS synth tests.") + + let fixtures = try LuxTtsFixtures.load() + let text = fixtures.texts[fixtures.e2e.textIndex] + + // Reconstruct the prompt clip from the 24 kHz fixture waveform so the + // test does not depend on files outside the repo. 16-bit quantization + // perturbs the mel at ~1e-4 — irrelevant for the stat gates below. + let promptSamples = try LuxTtsFixtures.loadFloats("prompt_24k_f32le.bin") + let promptURL = FileManager.default.temporaryDirectory + .appendingPathComponent("luxtts_e2e_prompt_\(UUID().uuidString).wav") + let promptWav = try AudioWAV.data( + from: promptSamples, + sampleRate: Double(LuxTtsConstants.melSampleRate), + normalize: false) + try promptWav.write(to: promptURL) + defer { try? FileManager.default.removeItem(at: promptURL) } + + let manager = try await LuxTtsManager.downloadAndCreate() + let isReady = await manager.isAvailable + XCTAssertTrue(isReady, "Manager did not become available after initialize()") + + let result = try await manager.synthesize( + phonemes: text.phonemeString, + promptAudio: promptURL, + promptPhonemes: fixtures.prompt.phonemeString, + speed: Float(fixtures.e2e.speed), + seed: UInt64(fixtures.e2e.seed)) + + // Frame accounting must be identical to Python (pure integer math + // over fixture-pinned token ids and mel frame counts). + XCTAssertEqual(result.sampleRate, 48000) + XCTAssertEqual(result.promptFrames, fixtures.prompt.melFrames) + XCTAssertEqual(result.featuresLength, fixtures.e2e.featuresLen) + XCTAssertEqual(result.generatedFrames, fixtures.e2e.genFrames) + XCTAssertEqual(result.samples.count, fixtures.e2e.wavSamples) + + // Loudness within 1 dB of the Python CoreML pipeline, and non-silent. + let sumSquares = result.samples.reduce(Double(0)) { $0 + Double($1) * Double($1) } + let rms = (sumSquares / Double(result.samples.count)).squareRoot() + XCTAssertGreaterThan(rms, 0.01, "output is (near-)silent") + let dbDelta = 20.0 * log10(rms / fixtures.e2e.rms) + XCTAssertLessThan(abs(dbDelta), 1.0, "RMS off by \(dbDelta) dB vs Python fixture") + + await manager.cleanup() + } + + func testRawTextSynthesisRequiresInitialize() async throws { + // No models needed: the raw-text overload phonemizes in-process + // (bundled G2P, no download) and must still fail loudly with + // `.notInitialized` before touching the pipeline. + let manager = LuxTtsManager() + do { + _ = try await manager.synthesize( + text: "hello", + promptAudio: URL(fileURLWithPath: "/nonexistent.wav"), + promptText: "hello") + XCTFail("expected notInitialized") + } catch let error as LuxTtsError { + guard case .notInitialized = error else { + XCTFail("expected notInitialized, got \(error)") + return + } + } + } +} diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsFixtures.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsFixtures.swift new file mode 100644 index 00000000..bcc0a96c --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsFixtures.swift @@ -0,0 +1,145 @@ +import Foundation +import XCTest + +/// Typed access to the Python-generated LuxTTS parity fixtures +/// (`Resources/luxtts_fixtures.json` + binary companions), produced by +/// `mobius/models/tts/zipvoice/coreml/dump_swift_fixtures.py`. +struct LuxTtsFixtures: Decodable { + + struct Expansion: Decodable { + let tokensLen: Int + let featuresLen: Int + let avgTokenDuration: Int + let tokensIndexLen: Int + let tokensIndex: [Int] + + enum CodingKeys: String, CodingKey { + case tokensLen = "tokens_len" + case featuresLen = "features_len" + case avgTokenDuration = "avg_token_duration" + case tokensIndexLen = "tokens_index_len" + case tokensIndex = "tokens_index" + } + } + + struct Prompt: Decodable { + let transcript: String + let rmsPreNorm: Double + let targetRms: Double + let wav24kSamples: Int + let melFrames: Int + let melDim: Int + let melFirst3Frames: [[Double]] + let phonemeString: String + let tokenIds: [Int] + + enum CodingKeys: String, CodingKey { + case transcript + case rmsPreNorm = "rms_pre_norm" + case targetRms = "target_rms" + case wav24kSamples = "wav_24k_samples" + case melFrames = "mel_frames" + case melDim = "mel_dim" + case melFirst3Frames = "mel_first3_frames" + case phonemeString = "phoneme_string" + case tokenIds = "token_ids" + } + } + + struct Text: Decodable { + let text: String + let phonemeString: String + let tokenIds: [Int] + let catTokensLen: Int + let featuresLenSpeed1: Int + let expansion: Expansion + + enum CodingKeys: String, CodingKey { + case text + case phonemeString = "phoneme_string" + case tokenIds = "token_ids" + case catTokensLen = "cat_tokens_len" + case featuresLenSpeed1 = "features_len_speed1" + case expansion + } + } + + struct MiniTrajectory: Decodable { + let x0: [Double] + let vSteps: [[Double]] + let xFinal: [Double] + + enum CodingKeys: String, CodingKey { + case x0 + case vSteps = "v_steps" + case xFinal = "x_final" + } + } + + struct Solver: Decodable { + let numSteps: Int + let tShift: Double + let timesteps: [Double] + let miniTrajectory: MiniTrajectory + + enum CodingKeys: String, CodingKey { + case numSteps = "num_steps" + case tShift = "t_shift" + case timesteps + case miniTrajectory = "mini_trajectory" + } + } + + struct E2E: Decodable { + let textIndex: Int + let seed: Int + let speed: Double + let featuresLen: Int + let genFrames: Int + let wavSamples: Int + let wavSeconds: Double + let rms: Double + + enum CodingKeys: String, CodingKey { + case textIndex = "text_index" + case seed + case speed + case featuresLen = "features_len" + case genFrames = "gen_frames" + case wavSamples = "wav_samples" + case wavSeconds = "wav_seconds" + case rms + } + } + + let prompt: Prompt + let texts: [Text] + let solver: Solver + let e2e: E2E + + // MARK: - Loading + + static func resourceURL(_ name: String) throws -> URL { + guard + let url = Bundle.module.url( + forResource: name, withExtension: nil, subdirectory: "Resources") + else { + throw XCTSkip("LuxTts fixture resource missing: \(name)") + } + return url + } + + static func load() throws -> LuxTtsFixtures { + let data = try Data(contentsOf: resourceURL("luxtts_fixtures.json")) + return try JSONDecoder().decode(LuxTtsFixtures.self, from: data) + } + + /// Read a little-endian Float32 binary fixture. + static func loadFloats(_ name: String) throws -> [Float] { + let data = try Data(contentsOf: resourceURL(name)) + precondition(data.count % 4 == 0, "\(name) size not a multiple of 4") + return data.withUnsafeBytes { raw in + Array(raw.bindMemory(to: Float.self)) + } + } +} diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsG2pTests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsG2pTests.swift new file mode 100644 index 00000000..a942e7b6 --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsG2pTests.swift @@ -0,0 +1,125 @@ +import XCTest + +@testable import FluidAudio + +/// Espeak-parity G2P tests. Expected strings are the espeak-ng oracle +/// output (piper_phonemize en-us via EmiliaTokenizer, i.e. the exact +/// frontend LuxTTS was trained on), generated with +/// `mobius/models/tts/zipvoice/coreml/g2p/validate.py dump-oracle`. +/// +/// Corpus-level gates (1,000 sentences: 99.6% sentence exact match, +/// 0.01% token edit rate) are enforced by the reproducible harness in +/// the mobius worktree; these tests pin representative behaviors. +final class LuxTtsG2pTests: XCTestCase { + + private static let g2p: LuxTtsG2p = { + do { + return try LuxTtsG2p() + } catch { + fatalError("cannot load bundled G2P resources: \(error)") + } + }() + + private func assertPhonemes( + _ text: String, _ expected: String, + file: StaticString = #filePath, line: UInt = #line + ) { + XCTAssertEqual( + Self.g2p.phonemize(text: text), expected, file: file, line: line) + } + + // MARK: - Phase-1 fixture sentences + + func testFixtureSentence1() { + assertPhonemes( + "The quick brown fox jumps over the lazy dog, and honestly, it felt great.", + "ðə kwˈɪk bɹˈaʊn fˈɑːks dʒˈʌmps ˌoʊvɚ ðə lˈeɪzi dˈɑːɡ, ænd ˈɑːnɪstli, ɪt fˈɛlt ɡɹˈeɪt.") + } + + func testFixtureSentence2CamelCase() { + assertPhonemes( + "FluidAudio runs speech models locally on Apple silicon, no cloud required.", + "flˈuːɪd ˈɔːdɪˌoʊ ɹˈʌnz spˈiːtʃ mˈɑːdəlz lˈoʊkəli ˌɔn ˈæpəl sˈɪlɪkən, nˈoʊ klˈaʊd ɹᵻkwˈaɪɚd.") + } + + func testFixtureSentence3PromptTranscript() { + assertPhonemes( + "Quick brown fox jumps over the lazy dog and honestly it felt great.", + "kwˈɪk bɹˈaʊn fˈɑːks dʒˈʌmps ˌoʊvɚ ðə lˈeɪzi dˈɑːɡ ænd ˈɑːnɪstli ɪt fˈɛlt ɡɹˈeɪt.") + } + + // MARK: - Numbers / normalization (upstream ZipVoice normalizer parity) + + func testNumbersTimeOrdinalCurrency() { + assertPhonemes( + "The meeting starts at 3:45 PM on March 21st, and it costs $12.50.", + "ðə mˈiːɾɪŋ stˈɑːɹts æt θɹˈiː: fˈɔːɹɾifˈaɪv pˌiːˈɛm ˌɔn mˈɑːɹtʃ twˈɛntifˈɜːst, " + + "ænd ɪt kˈɔsts twˈɛlv dˈɑːlɚz, fˈɪfti sˈɛnts.") + } + + func testAbbreviationAndHomographs() { + // Dr. -> doctor; "read" defaults to present, stays present after + // pronoun "I" ($verbf window); "the record" selects the noun form. + assertPhonemes( + "Dr. Smith read the record; I read it yesterday.", + "dˈɑːktɚ.smˈɪθ ɹˈiːd ðə ɹˈɛkɚd; aɪ ɹˈiːd ɪt jˈɛstɚdˌeɪ.") + } + + func testNormalizerYearAndCardinal() { + XCTAssertEqual( + LuxTtsEnglishNormalizer.normalize("born in 1855, moved in 2007"), + "born in eighteen fifty-five , moved in two thousand seven ") + XCTAssertEqual(LuxTtsEnglishNormalizer.cardinalWords(123), "one hundred twenty-three") + XCTAssertEqual(LuxTtsEnglishNormalizer.ordinalWords("42nd"), "forty-second") + } + + // MARK: - espeak clause behaviors + + func testWeakFormsAndMerges() { + // "in the" merges without a space; "the" -> ðɪ before vowels; + // capital I is the unstressed pronoun (aɪ), lowercase i the letter. + assertPhonemes("in the house", "ɪnðə hˈaʊs") + assertPhonemes("the apple", "ðɪ ˈæpəl") + assertPhonemes("I want to eat", "aɪ wˈɔnt tʊ ˈiːt") + assertPhonemes("I want to go", "aɪ wˈɔnt tə ɡˈoʊ") + } + + func testStrendStressResolution() { + // $strend2: "over" is fully stressed only when followed by + // unstressed words (with linking-r before a vowel). + assertPhonemes("jump over it", "dʒˈʌmp ˈoʊvɚɹ ɪt") + assertPhonemes("jumps over the lazy dog", "dʒˈʌmps ˌoʊvɚ ðə lˈeɪzi dˈɑːɡ") + } + + func testAllCapsSpellOut() { + // "FBI" spells out to ˌɛfbˌiːˈaɪ; the leading vowel of the F=ɛf letter + // triggers the before-vowel weak form of "the" (ðɪ, not ðə) — matching + // the espeak oracle (piper_phonemize en-us via EmiliaTokenizer). + assertPhonemes("the FBI called", "ðɪ ˌɛfbˌiːˈaɪ kˈɔːld") + } + + func testPossessiveFallback() { + // "John's" resolves via the possessive rule when absent from the + // lexicon row set (voiced final -> z). + let phonemes = Self.g2p.phonemize(text: "John's dog") + XCTAssertTrue(phonemes.hasPrefix("dʒˈɑːnz"), "got \(phonemes)") + } + + // MARK: - Token id mapping (fixture ids from the espeak oracle) + + func testFixtureSentence1TokenIds() throws { + let expected = [ + 41, 59, 3, 23, 35, 120, 74, 23, 3, 15, 88, 120, 14, 100, 26, 3, + 19, 120, 51, 122, 23, 31, 3, 17, 108, 120, 102, 25, 28, 31, 3, + 121, 27, 100, 34, 60, 3, 41, 59, 3, 24, 120, 18, 74, 38, 21, 3, + 17, 120, 51, 122, 66, 8, 3, 39, 26, 17, 3, 120, 51, 122, 26, 74, + 31, 32, 24, 21, 8, 3, 74, 32, 3, 19, 120, 61, 24, 32, 3, 66, 88, + 120, 18, 74, 32, 10, + ] + let tokensURL = try LuxTtsFixtures.resourceURL("tokens.txt") + let tokenizer = try LuxTtsTokenizer(tokensFileURL: tokensURL) + let phonemes = Self.g2p.phonemize( + text: "The quick brown fox jumps over the lazy dog, and honestly, it felt great.") + XCTAssertEqual(tokenizer.tokenIds(phonemes: phonemes), expected) + } +} diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsMelExtractorTests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsMelExtractorTests.swift new file mode 100644 index 00000000..c11cf8b8 --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsMelExtractorTests.swift @@ -0,0 +1,42 @@ +import XCTest + +@testable import FluidAudio + +/// Pins `LuxTtsMelExtractor` against the Python `VocosFbank` fixture mel of +/// the prompt clip (24 kHz, n_fft 1024, hop 256, 100 mels, log + ×0.1). +final class LuxTtsMelExtractorTests: XCTestCase { + + func testFrameCountMatchesLhotse() { + let extractor = LuxTtsMelExtractor() + // lhotse compute_num_frames: (n + hop/2) / hop. + XCTAssertEqual(extractor.frameCount(sampleCount: 103936), 406) + XCTAssertEqual(extractor.frameCount(sampleCount: 256), 1) + XCTAssertEqual(extractor.frameCount(sampleCount: 127), 0) + XCTAssertEqual(extractor.frameCount(sampleCount: 128), 1) + } + + func testPromptMelMatchesFixture() throws { + let fixtures = try LuxTtsFixtures.load() + let audio = try LuxTtsFixtures.loadFloats("prompt_24k_f32le.bin") + XCTAssertEqual(audio.count, fixtures.prompt.wav24kSamples) + + let extractor = LuxTtsMelExtractor() + let mel = extractor.extract(audio: audio) + XCTAssertEqual(mel.count, fixtures.prompt.melFrames) + XCTAssertEqual(mel.first?.count, fixtures.prompt.melDim) + + // Fixture mel is feat-scaled (×0.1); the extractor output is not. + let reference = try LuxTtsFixtures.loadFloats("prompt_mel_f32le.bin") + XCTAssertEqual(reference.count, mel.count * fixtures.prompt.melDim) + + var maxAbsDiff: Float = 0 + for frame in 0.. LuxTtsTokenizer { + try LuxTtsTokenizer(tokensFileURL: LuxTtsFixtures.resourceURL("tokens.txt")) + } + + func testTokenTableParsing() throws { + let tokenizer = try makeTokenizer() + XCTAssertEqual(tokenizer.vocabSize, 360) + XCTAssertEqual(tokenizer.padId, 0) + XCTAssertEqual(tokenizer.tokenToId["_"], 0) + XCTAssertEqual(tokenizer.tokenToId[" "], 3) + XCTAssertEqual(tokenizer.tokenToId["ˈ"], 120) + // Multi-character pinyin entries survive the first-tab-only split. + XCTAssertNotNil(tokenizer.tokenToId["zh0"]) + } + + func testPromptTokenIdsMatchFixture() throws { + let fixtures = try LuxTtsFixtures.load() + let tokenizer = try makeTokenizer() + XCTAssertEqual( + tokenizer.tokenIds(phonemes: fixtures.prompt.phonemeString), + fixtures.prompt.tokenIds) + } + + func testTextTokenIdsMatchFixture() throws { + let fixtures = try LuxTtsFixtures.load() + let tokenizer = try makeTokenizer() + for text in fixtures.texts { + XCTAssertEqual( + tokenizer.tokenIds(phonemes: text.phonemeString), + text.tokenIds, + "token ids diverged for: \(text.text)") + } + } + + func testOovScalarsAreSkipped() throws { + let tokenizer = try makeTokenizer() + // '€' is not in tokens.txt; the surrounding tokens must be unaffected + // (EmiliaTokenizer skips OOV entries). + XCTAssertEqual( + tokenizer.tokenIds(phonemes: "a€b"), + tokenizer.tokenIds(phonemes: "ab")) + } +} diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsVariantTests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsVariantTests.swift new file mode 100644 index 00000000..97ab2004 --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsVariantTests.swift @@ -0,0 +1,142 @@ +import CoreML +import XCTest + +@testable import FluidAudio + +/// Wiring tests for the per-platform graph variant (`gpu/` on macOS, +/// `ane/` on iOS). These are model-independent: they pin the compute-unit +/// policy, the download file-set (so macOS never pulls `ane/` and iOS never +/// pulls `gpu/`), and the fixed I/O shapes the synthesizer packs — which the +/// `ane/` and `gpu/` FmDecoder graphs share byte-for-byte. +final class LuxTtsVariantTests: XCTestCase { + + // MARK: - Compute-unit policy + + func testAneVariantUsesNeuralEngine() { + XCTAssertEqual( + LuxTtsModelStore.encoderDecoderComputeUnits( + variant: ModelNames.LuxTts.aneVariant, override: nil), + .cpuAndNeuralEngine) + } + + func testGpuVariantUsesGpu() { + XCTAssertEqual( + LuxTtsModelStore.encoderDecoderComputeUnits( + variant: ModelNames.LuxTts.gpuVariant, override: nil), + .cpuAndGPU) + } + + func testExplicitOverrideWinsForBothVariants() { + for variant in [ModelNames.LuxTts.gpuVariant, ModelNames.LuxTts.aneVariant] { + XCTAssertEqual( + LuxTtsModelStore.encoderDecoderComputeUnits(variant: variant, override: .cpuOnly), + .cpuOnly, + "override must win for variant \(variant)") + } + } + + // MARK: - Download file-set (variant isolation) + + func testAneVariantRequiresOnlyAneModels() { + let files = ModelNames.LuxTts.requiredFiles(variant: ModelNames.LuxTts.aneVariant) + XCTAssertTrue(files.contains("ane/TextEncoder.mlmodelc")) + XCTAssertTrue(files.contains("ane/FmDecoder.mlmodelc")) + // iOS must never drag in the gpu graph. + XCTAssertFalse(files.contains { $0.hasPrefix("gpu/") }) + } + + func testGpuVariantRequiresOnlyGpuModels() { + let files = ModelNames.LuxTts.requiredFiles(variant: ModelNames.LuxTts.gpuVariant) + XCTAssertTrue(files.contains("gpu/TextEncoder.mlmodelc")) + XCTAssertTrue(files.contains("gpu/FmDecoder.mlmodelc")) + // macOS must never drag in the ane graph. + XCTAssertFalse(files.contains { $0.hasPrefix("ane/") }) + } + + func testSharedAssetsAreRequiredForBothVariants() { + let shared: Set = [ + ModelNames.LuxTts.tokensFile, + ModelNames.LuxTts.configFile, + ModelNames.LuxTts.vocoder282File, + ModelNames.LuxTts.vocoder555File, + ] + for variant in [ModelNames.LuxTts.gpuVariant, ModelNames.LuxTts.aneVariant] { + let files = ModelNames.LuxTts.requiredFiles(variant: variant) + XCTAssertTrue( + shared.isSubset(of: files), + "variant \(variant) is missing shared assets \(shared.subtracting(files))") + } + } + + // MARK: - FmDecoder I/O contract (shared ane/gpu external signature) + + /// The synthesizer packs the FmDecoder inputs at these fixed shapes; the + /// compiled `ane/` and `gpu/` graphs both declare exactly these. A drift + /// here (e.g. a bucket change) would silently mispack the ANE decoder. + func testFmDecoderContractShapes() throws { + let featDim = LuxTtsConstants.featDim + let maxFrames = LuxTtsConstants.maxFrames + let maxTokens = LuxTtsConstants.maxTokens + XCTAssertEqual(featDim, 100) + XCTAssertEqual(maxFrames, 1024) + XCTAssertEqual(maxTokens, 256) + + // x / text_condition / speech_condition: (1, 1024, 100) + let x = try MLMultiArray(shape: [1, maxFrames, featDim].map { NSNumber(value: $0) }, dataType: .float32) + XCTAssertEqual(x.shape.map { $0.intValue }, [1, 1024, 100]) + // padding_mask: (1, 1024) + let mask = try MLMultiArray(shape: [1, maxFrames].map { NSNumber(value: $0) }, dataType: .float32) + XCTAssertEqual(mask.shape.map { $0.intValue }, [1, 1024]) + // t / guidance_scale: (1) + let scalar = try MLMultiArray(shape: [NSNumber(value: 1)], dataType: .float32) + XCTAssertEqual(scalar.shape.map { $0.intValue }, [1]) + // tokens: (1, 256) int32 + let tokens = try MLMultiArray(shape: [1, maxTokens].map { NSNumber(value: $0) }, dataType: .int32) + XCTAssertEqual(tokens.shape.map { $0.intValue }, [1, 256]) + XCTAssertEqual(tokens.dataType, .int32) + } + + /// The FmDecoder `v` output is `(1, 1024, 100)` fp32 row-major, and CoreML + /// stride-pads the last dim on some backends. This mirrors the + /// synthesizer's `copyRows` unpack: read `featuresLength` rows of + /// `featDim` honoring the row stride, so an ANE stride-padded output is + /// de-padded correctly. Regression guard for the ANE unpack path. + func testStridePaddedOutputUnpack() throws { + let featDim = LuxTtsConstants.featDim + let rows = 8 + let paddedRowLen = 112 // e.g. ANE/GPU pad 100 → 112 + + let backing = try MLMultiArray( + shape: [1, NSNumber(value: rows), NSNumber(value: paddedRowLen)], + dataType: .float32) + // Fill each active [row, d] cell with a unique sentinel; pad region 0. + backing.withUnsafeMutableBufferPointer(ofType: Float.self) { buf, _ in + for r in 0..