fix(sortformer): streaming mel emits phantom frames, drifting all timestamps late

[predict-woo]

Problem

The streaming preprocessor (addAudio / process) center-pads the buffered audio on every computeFlatTransposed call, but the samplesConsumed inversion only accounts for one pad. Each preprocess call therefore emits frames covering nFFT - (nFFT - melWindow)/2 = 272 samples (17 ms) more timeline than the audio it consumes.

With real-time feeding that is roughly one call per 6-frame chunk, so the finalized frame stream runs ~3.5% faster than the audio, and every frame-derived timestamp (speaker turns, segment boundaries) lags further behind the audio as the session runs. Feeding the whole file in a single addAudio call only pads once, which is why batch-fed tests and processComplete (which uses SortformerFeatureLoader) never hit this. It is the same padding-accounting family as #441 / #422 / #444, but silent: there is no static shape to trip a CoreML error, the feature buffer just absorbs the extra frames. The closest acknowledgment was the "known architectural limitation" note in SortformerTests.

I hit this in a real app: on a 2.5 minute two-speaker recording, the finalized speaker turns were stamped 2.7 s late at the first turn and 4.5 s late a minute later, so each turn's opening seconds were attributed to the previous speaker, while the tentative (real-time) predictions looked fine.

Same demo script, same audio, against current main and this branch:

Fix

Emit the session-global, batch-equivalent mel stream incrementally instead of re-padding per call:

• audioBuffer always keeps nFFT/2 samples of left context, seeded with the batch left pad at reset. Mid-stream frames are computed from real samples only (.prePadded + expectedFrameCount), consuming exactly count * melStride samples, so feeding granularity cannot change frame count or values.
• A frame is emitted once its window [k*hop - win/2, k*hop + win/2) is covered by received samples.
• finalizeSession (and speaker enrollment, which feeds a complete clip) appends the batch right pad and emits the trailing frames, reaching the exact .center-mode frame count. The pad is a preemphasis-cancelling decay so the in-place preemphasis filter reproduces the batch pad's exact zeros.

Streaming output now matches computeFlatTransposed(paddingMode: .center) of the whole session frame-for-frame at any feeding granularity, so the limitation note is gone too. No public API changes.

Test plan

• New SortformerStreamingMelTests: identical chunk sequences and frame counts across feeding granularities (160 / 1600 / 4093 / 16000 / one-shot), finalized frame count matches the batch formula, streamed values match batch center-padded mel including the padded tail, and clean restart after reset().
• Existing Sortformer, enrollment, timeline, and streaming-integration suites pass unchanged.
• Verified on the real 154 s recording above: finalized speaker turns now land within one frame of the tentative detections.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[Alex-Wengg]

Do you have a sample wav file or steps to reproduce this issue

[Alex-Wengg]

from ai

1. Failed/thrown enrollment permanently bricks the diarizer. enrollSpeaker calls padAndEmitRemainingMelLocked() (setting melInputExhausted = true), but the guard didProcess early return (~line 332) is the one exit path that never calls resetMelStreamLocked() — unlike the sibling paths at 358/380. Any enrollment clip under ~1.02 s (fewer than the 104 mel frames one chunk needs) hits it, and every later addAudio/process/finalizeSession becomes a silent no-op until an explicit reset(). A CancellationError (from Task.checkCancellation) or CoreML throw mid-enrollment skips the cleanup the same way, so adding a reset before the return nil isn't enough — please move the cleanup into a defer (or a shared helper; the 5-line cleanup block is already duplicated verbatim at ~355-360 and ~377-381, so extracting it fixes both problems at once). On main, these paths left stale buffers but streaming kept working.

2. Post-exhaustion addAudio grows audioBuffer unboundedly. Once the latch is set (via finalizeSession or the failed-enrollment path above), all addAudio overloads still append samples and bump _realSamplesReceived, while preprocessAudioToFeaturesLocked bails and nothing ever drains the buffer — ~230 MB/hour at 16 kHz with zero output and no log. Pre-PR, post-finalize audio was still consumed, so buffers stayed bounded. An early return (with a one-time warning log) in addAudio/process when melInputExhausted is set would cover it.

Two smaller asks while you're in there:

• Fold _realSamplesReceived = 0 into resetMelStreamLocked(). It's the only piece of mel-stream state the reset skips, yet it now drives the frame math — a future bare call with a stale count makes emitMelFramesLocked index audioBuffer[count * melStride - 1] into the 256-sample seed and trap. Safe today: enrollSpeaker overwrites it after the call, and the method never reads it.
• mel.prefix(melLength * config.melFeatures) is a no-op for this instance (default padTo clamps to 1, so computeFlatTransposed returns exactly count frames) — a plain append with assert(melLength == count) would surface a future contract break instead of silently truncating it.

[predict-woo]

Yes! I made a small repro with Claude: https://github.com/predict-woo/fluidaudio-streaming-drift-repro

It comes with a 72-second wav where two macOS say voices take turns talking, so we know exactly when each speaker starts: 18.4 s, 36.8 s, and 54.7 s.

The script runs this file through the diarizer twice. First it hands over the whole file at once, then it feeds the same file in small 100 ms pieces, like a live microphone would. The results should be identical, but on current main they aren't:

• Whole file at once: turns detected at 18.5 / 36.9 / 54.7 s. Basically perfect.
• 100 ms pieces: turns detected at 19.1 / 38.2 / 56.7 s. Each one is later than the last, up to 2 seconds off, and it keeps getting worse the longer the audio runs. The diarizer also reports 75 s of output for 72.5 s of audio.

Just run swift run -c release repro and it prints both results side by side with a PASS/FAIL at the end. On this PR's branch the same repro prints PASS, with both runs giving identical timestamps.

[predict-woo]

All four should be fixed in 4ac06f1. The enrollment cleanup now lives in a defer so the short-clip return and thrown errors can't leave the stream latched anymore (that also got rid of the duplicated cleanup blocks), audio after finalize is dropped with a one-time warning instead of piling up in the buffer, _realSamplesReceived is reset inside resetMelStreamLocked() now, and the prefix call became a plain append with an assert. I also added a test for the enrollment case, and it does fail on the previous commit. Small heads up: I had to make it skip via XCTSkip when models can't download, because the usual XCTExpectFailure wrapper was quietly swallowing the real failure.