Skip to content

feat(ffi): add normalize_sentence_lang for multi-language ITN (#83)#84

Merged
Alex-Wengg merged 1 commit into
mainfrom
feat/itn-normalize-sentence-lang
Jul 23, 2026
Merged

feat(ffi): add normalize_sentence_lang for multi-language ITN (#83)#84
Alex-Wengg merged 1 commit into
mainfrom
feat/itn-normalize-sentence-lang

Conversation

@Alex-Wengg

Copy link
Copy Markdown
Member

Closes #83.

Problem

The library covered all 7 languages for single-expression ITN (normalize_with_lang) and for sentence-level TN (tn_normalize_sentence_lang), but sentence-level ITN was English-only. Non-English dictation cleanup (Parakeet STT post-processing) got no number formatting, because callers only have full sentences — not the clean single expressions normalize_with_lang expects.

Change

Adds the symmetric ITN sentence API across every binding layer:

Layer Added
Rust lib normalize_sentence_lang(input, lang) + normalize_sentence_with_max_span_lang(...)
C FFI nemo_normalize_sentence_lang(input, lang) + header decl
Swift NemoTextProcessing.normalizeSentence(_:language:)
WASM/npm normalizeSentenceLang(input, lang)

Two tagger architectures

The interesting part is that ITN languages split into two families, dispatched in normalize_sentence_with_max_span_lang:

  • en/fr/de/es — expression-level taggers that expect a clean span. These are scanned with the existing sliding window (sentence_loop) via a new parse_span_lang, which delegates to the per-language try_*_taggers helpers (already ordered by priority).
  • hi/ja/zh — scanning taggers that already replace number spans in place across the whole input, so normalize_with_lang is the sentence-level entry point for them and max_span_tokens doesn't apply.
  • Unrecognized codes (and "") fall back to English sentence mode.

This is why the TN side's single uniform macro couldn't just be copied — the ITN language module sets are non-uniform (e.g. ja has no whitelist/money/measure, hi adds address).

The headline behavior the issue asked for:

// whole-input normalize_with_lang leaves this untouched (not a clean expression)
normalize_sentence_lang("j'ai vingt et un ans", "fr")            // "j'ai 21 ans"
normalize_sentence_lang("j'ai vingt et un ans et trente chiens", "fr") // "j'ai 21 ans et 30 chiens"

Tests

  • tests/multilang_itn_tests.rs — all 7 languages, multi-expression sentences, English/unknown fallback, empty/whitespace, max_span behavior.
  • FFI unit tests (nemo_normalize_sentence_lang incl. null handling).
  • Rust doctest on the public fn.
  • Node WASM smoke assertion.

Full suite green (cargo test, --features ffi, --doc), cargo fmt --check clean, release+ffi build exports the symbol.

Known limitation (pre-existing, not introduced here)

The Spanish cardinal tagger over-eagerly joins "y"normalize_with_lang("diez y veinte", "es") already returns "30" on main. Sentence-scanning makes this reachable more often (e.g. "manzanas y diez peras" drops the "y"). This is a tagger-quality issue in itn::es::cardinal, independent of the FFI wiring; worth a separate follow-up rather than expanding this PR's scope. Test assertions avoid "y"-joined Spanish cases.

The library covered all 7 languages for single-expression ITN
(normalize_with_lang) and for sentence-level TN
(tn_normalize_sentence_lang), but sentence-level ITN was English-only.
Non-English dictation (Parakeet STT post-processing) got no number
formatting because callers only have full sentences, not clean
expressions.

Add the symmetric ITN sentence API across every binding layer:

- lib: normalize_sentence_lang / normalize_sentence_with_max_span_lang.
  Two tagger architectures are dispatched: en/fr/de/es use
  expression-level taggers scanned with the shared sliding window
  (sentence_loop + new parse_span_lang), while hi/ja/zh taggers already
  scan the whole sentence in place, so normalize_with_lang is the
  sentence entry point for them. Unrecognized codes fall back to English.
- ffi: nemo_normalize_sentence_lang + header declaration.
- swift: NemoTextProcessing.normalizeSentence(_:language:).
- wasm: normalizeSentenceLang + node smoke assertion.

Tests: multilang_itn_tests.rs (all 7 langs, multi-expression, fallback,
edge cases), FFI unit tests, doctest. Full suite green, fmt clean.

Known limitation (pre-existing, not introduced here): the Spanish
cardinal tagger over-eagerly joins "y" (e.g. "diez y veinte" -> "30"),
which sentence-scanning now surfaces more often. Tracked separately as a
tagger-quality follow-up.
@Alex-Wengg
Alex-Wengg merged commit 4f1f3e9 into main Jul 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C FFI: add nemo_normalize_sentence_lang for multi-language ITN

1 participant