Add NVIDIA parakeet-unified-en-0.6b (fixed-window streaming, NPU)

[Alex-Wengg]

Adds the NVIDIA parakeet-unified-en-0.6b model (English, unified offline/streaming FastConformer-RNNT) to the eddy::nemotron backend.

What's new

• Window-streaming inference path — parakeet-unified is stateless (no caches, no prompt) and streams via a fixed [left|chunk|right] attention window. The backend auto-detects this from metadata and runs a sliding-window decode: featurize the window → static-shape encoder → keep the chunk's encoder frames → greedy RNNT, carrying LSTM state across windows. Distinct from Nemotron's cache-aware loop; reuses the decoder/joint and the NPU BitwiseNot→LogicalNot fix.
• Per-feature mel normalization in MelFeaturizer (NeMo per_feature: mean/std per bin over valid frames), gated by a feature_normalize metadata flag. parakeet-unified exports normalized features; Nemotron does not (defaults off — unchanged).
• Encoder cache/prompt auto-detection generalized: cache inputs detected (has_cache); cache shapes required only when present; the featurizer geometry guard is scoped to the cache-aware path.
• Config: parakeet-unified → FluidInference/parakeet-unified-en-0.6b-ov (streaming/fp16, static shapes for NPU).

Static shapes → NPU

The offline (full-context) encoder is dynamic-length and won't compile on the NPU; the fixed-window streaming export (mel[1,128,769] → encoded[1,1024,97]) is static and compiles + runs correctly on NPU.

Validation (Intel NPU)

• End-to-end transcript correct: "…the romantic and fascinating town of Sintra…"
• FLEURS en_us samples: WER 5.9% / CER 1.8%, RTFx ~11x.
• Nemotron multilingual + English regression-checked: unchanged.

IR produced by the mobius OpenVINO export (models/stt/parakeet-unified-en-0.6b/openvino/export_openvino.py) and published to the model repo (fp16/ offline + streaming/fp16/ NPU).

🤖 Generated with Claude Code

[claude]

Code Review

This PR adds parakeet-unified-en-0.6b (stateless FastConformer-RNNT) to the eddy::nemotron backend via a new fixed-window sliding-decode path, and extends MelFeaturizer with optional per-feature normalization. The architecture is clean and the auto-detection of model type from metadata is a good generalisation. A few correctness issues in the new path are worth addressing before merge.

---

Bug: per-feature normalization includes zero-padded frames at window boundaries

src/models/nemotron/nemotron_openvino.cpp line 392

I.featurizer->compute(window.data(), window.size(), static_cast<int>(window.size()),
                      mel_scratch, t_mel);

window.size() is passed as both n (total samples) and valid_samples. Because the window is zero-padded at its left edge for the first chunk and at its right edge for the last chunk, valid_frames ends up covering silence frames. The per-feature normalization then computes mean/std over those zero-frames, pulling the mean negative and deflating the std — producing statistics that diverge from what NeMo's preprocessor computes (which excludes padding).

The fix is to compute the real non-padded count:

// How many samples in [start, start+win_s) actually come from pcm[0..N)?
const long real_lo  = std::max(0L, -start);
const long real_hi  = std::min(win_s, N - start);
const int  valid_n  = static_cast<int>(std::max(0L, real_hi - real_lo));
I.featurizer->compute(window.data(), window.size(), valid_n, mel_scratch, t_mel);

Middle windows are unaffected (all samples are real audio); only boundary windows carry the bug. Given WER 5.9% was reported on interior speech, the true WER at audio boundaries is likely higher.

---

Bug: infinite loop when chunk_enc is absent from metadata

src/models/nemotron/nemotron_openvino.cpp line 384

for (long pos = 0; pos < N; pos += chunk_s) {

chunk_enc defaults to 0 (line 156) and is only set if context_encoder_frames in the metadata contains a "chunk" key. If that key is absent or the metadata is malformed, chunk_s == 0 and pos never advances — the process hangs forever. The featurizer geometry guard that would have caught a shape mismatch is explicitly skipped for window_streaming models (line 285), so there is no secondary safety net.

Add a startup guard in ensure_compiled():

if (impl_->window_streaming && impl_->chunk_enc <= 0)
    throw std::runtime_error("parakeet-unified metadata missing context_encoder_frames.chunk");

---

Bug: zero-size mel tensor when window_mel_frames is absent

src/models/nemotron/nemotron_openvino.cpp line 379

ov::Tensor mel_in(ov::element::f32, ov::Shape{1, bins, win_mel});

win_mel comes from m.value("window_mel_frames", 0). If the key is absent, win_mel == 0 and mel_in is a zero-sized tensor. The encoder infer() call then receives an input whose innermost dimension is 0, which either throws an OV shape-inference error or silently processes nothing. Same guard location as above:

if (impl_->window_streaming && impl_->window_mel_frames <= 0)
    throw std::runtime_error("parakeet-unified metadata missing window_mel_frames");

---

Missing featurizer geometry validation for the window-streaming path

src/models/nemotron/nemotron_openvino.cpp line 285

if (!impl_->window_streaming) {
    // probe: chunk_mel_frames+1 frames expected ...
}

The guard that validates the featurizer's framing matches the model geometry was correct for cache-aware encoders but has no equivalent for the new path. If window_mel_frames in metadata disagrees with what the featurizer actually produces for win_s samples, the mel is silently truncated or over-copied into mel_in (line 396–398) without any diagnostic. Since this is a static-shape NPU model the mismatch would corrupt every inference. Consider adding a similar probe for the window-streaming case:

if (impl_->window_streaming) {
    std::vector<float> probe(static_cast<size_t>(win_s), 0.0f);
    std::vector<float> mel_probe;
    size_t t_mel_probe = 0;
    impl_->featurizer->compute(probe.data(), probe.size(),
                               static_cast<int>(probe.size()), mel_probe, t_mel_probe);
    if (t_mel_probe != impl_->window_mel_frames)
        throw std::runtime_error("featurizer/window_mel_frames mismatch");
}

---

Minor: early-compile NPU cost before cache-shape validation (regression)

src/models/nemotron/nemotron_openvino.cpp ~line 247

The old code threw immediately upon reading metadata if cache shape keys were missing. The refactored code runs load_npu_safe() and compile_model() (potentially minutes of NPU compilation) before the moved check fires at line ~324. This is functionally correct — has_cache isn't knowable before compile — but a corrupted metadata.json for a Nemotron model now burns full compile time before producing an error. Low severity since it's a one-time startup failure, but worth a comment explaining why the check is post-compile.

---

Minor: lg[0] accessed before bounds check

src/models/nemotron/nemotron_openvino.cpp line 431

float best_score = lg[0];

vsz = logits.get_size() is computed two lines later but never checked before the first read. If the joint network returns an empty logits tensor (IR shape mismatch), this is undefined behaviour. A if (vsz == 0) break; before the argmax loop would make the failure mode explicit.

---

Summary: The first three are functional correctness bugs in the new path; the normalization one directly affects accuracy at audio boundaries. The others are hardening gaps. The core algorithmic approach (sliding-window RNNT with carried LSTM state, auto-detected from metadata) looks correct.

🤖 Generated with Claude Code

[claude]

Code Review

This PR adds the NVIDIA parakeet-unified-en-0.6b model to the eddy::nemotron backend via a new fixed-window sliding-encoder inference path and NeMo per-feature mel normalization. The overall design is solid — the metadata-driven auto-detection, geometry validation at startup, and NPU/BitwiseNot reuse are all well-executed. A few correctness gaps are worth addressing before merge.

---

🔴 CONFIRMED — window_streaming && has_cache have no mutual-exclusion guard

src/models/nemotron/nemotron_openvino.cpp:251 and :515

window_streaming is set from metadata (before compilation); has_cache is set from the compiled encoder's input ports (after compilation). There is no check that prevents both being true simultaneously. If they are, transcribe() dispatches to run_window_streaming(), which only calls set_tensor("mel", ...) and set_tensor("mel_length", ...) before encoder_req.infer(). The cache inputs (cache_channel, cache_time, cache_len) are never bound, so OpenVINO will throw or infer with stale data.

The fix is a single post-compile assertion in ensure_compiled():

if (impl_->window_streaming && impl_->has_cache) {
    throw std::runtime_error(
        "Model is both window-streaming and cache-aware; these are mutually exclusive: " +
        impl_->paths.metadata_json);
}

---

🔴 CONFIRMED — Missing h_out/c_out byte-size guard in run_window_streaming

src/models/nemotron/nemotron_openvino.cpp:477-480

The cache-aware path in transcribe() checks h_out.get_byte_size() != h.get_byte_size() before every LSTM-state memcpy and throws on mismatch (lines 702–707). run_window_streaming() performs the identical memcpy without this guard:

// Missing — present in transcribe() but absent here
std::memcpy(h.data<float>(), h_out.data<float>(), h.get_byte_size());
std::memcpy(c.data<float>(), c_out.data<float>(), c.get_byte_size());

If the decoder IR's actual output shape differs from what metadata specifies, this is a silent buffer over-read. The guard established in transcribe() should be applied here for consistency.

---

🟡 CONFIRMED — Per-feature normalization silently skipped for short tail windows

src/models/nemotron/nemotron_featurizer.cpp:201

The guard valid_frames >= 2 (required to avoid ddof=1 division by zero) also silently bypasses normalization when a tail window contains fewer than 2 valid frames — approximately when total_samples mod chunk_s < 320. For parakeet-unified with chunk_enc=8, subsampling=8, that's chunk_s=10240 and a ~3.1% hit rate (319/10240), guaranteed for some audio lengths. The encoder receives raw log-mel for those tail frames instead of the normalized features it was trained on, degrading accuracy for the last few hundred ms of affected clips. Short utterances (< ~640 ms) are always affected.

A simple mitigation: when valid_frames == 1, still normalize with the single frame's mean and a unit std (or clamp to valid_frames == 1 and use ddof=0). Alternatively, document the limitation and log it.

---

🟡 PLAUSIBLE — vsz == 0 guard in run_window_streaming not backported to transcribe()

src/models/nemotron/nemotron_openvino.cpp:682

run_window_streaming() defensively guards if (vsz == 0) break; before reading lg[0] (line 469, with a helpful comment). The identical inner-loop in transcribe()'s cache-aware path reads lg[0] unconditionally at line 684 — the guard exists in the new code but not in the old. While vsz == 0 is unlikely with static-shape NPU models, the PR added the guard for a reason and should backport it:

// In transcribe(), add before `int best = 0; float best_score = lg[0];`:
if (vsz == 0) break;

---

🟡 PLAUSIBLE — Cache tensors bound unconditionally in the non-window-streaming path regardless of has_cache

src/models/nemotron/nemotron_openvino.cpp:618-620

transcribe() always calls set_tensor("cache_channel", ...) / set_tensor("cache_time", ...) on the encoder request for any non-window-streaming model. has_cache is detected but never checked at this branch point. Any future stateless offline encoder that isn't window-streaming would trigger an OpenVINO set_tensor failure on non-existent ports. The guard is straightforward:

if (I.has_cache) {
    I.encoder_req.set_tensor("cache_channel", cache_channel);
    I.encoder_req.set_tensor("cache_time", cache_time);
    I.encoder_req.set_tensor("cache_len", cache_len);
}

---

🟡 PLAUSIBLE — blank_idx default (13087) may be wrong for parakeet-unified if metadata omits the key

src/models/nemotron/nemotron_openvino.cpp:232

impl_->blank_idx = m.value("blank_idx", 13087);

The default matches Nemotron's vocab size but is almost certainly wrong for parakeet-unified (English RNNT with a much smaller vocab). In run_window_streaming, if blank_idx never matches best, the inner sym-loop always runs to max_symbols_per_frame on every encoder frame, producing repeated spurious tokens. This depends on the external metadata.json artifact, but it's worth verifying that FluidInference/parakeet-unified-en-0.6b-ov/streaming/fp16/metadata.json explicitly includes blank_idx — and adding a validation assertion for the window-streaming path (e.g., blank_idx < vocab_size).

---

🔵 Cleanup — Greedy RNNT inner loop and piece lambda duplicated verbatim

src/models/nemotron/nemotron_openvino.cpp:454-501 vs :653-710

The token→decoder→joint→argmax→emit loop is copied nearly verbatim between run_window_streaming() and transcribe(). The piece lambda (lines 485 and 717) is byte-for-byte identical. Any future fix or feature (e.g. the vsz == 0 guard above, temperature, beam search) must be applied twice or the paths silently diverge. Consider extracting a greedy_rnnt_step() helper or at minimum hoisting piece into the anonymous namespace as a free function.

---

Summary

Severity	Count
🔴 CONFIRMED correctness	2
🟡 CONFIRMED accuracy / PLAUSIBLE correctness	4
🔵 Cleanup	1

The window_streaming && has_cache gap and the missing h_out/c_out guard are the highest priority — both are easily fixed and prevent silent failures on unexpected IR/metadata combinations.

🤖 Generated with Claude Code