[android] Validate on-device ASR runtime bench (#120) on Polish audio
Background
Issue #120's Android on-device ASR runtime spike (findings: docs/android-asr-runtime-bench.md,
PR #126) measured all four candidates — whisper.cpp, sherpa-onnx, Vosk, wav2vec2 — on a real
Pixel 7a, but every run so far is English-only. No Polish eval audio exists yet (blocked on
issues #63/#79/#30), so whisper.cpp's 89% E2E and sherpa-onnx's 87% E2E on-device numbers are not
yet confirmed to hold for Polish queries, even though both are advertised as multilingual models.
Why this matters
This project's stated goal is a bilingual (PL+EN) physicist assistant. The whole android runtime
decision in #120 was made on English-only evidence. If Polish accuracy is meaningfully worse
on-device — a real risk, since Whisper's per-language accuracy is known to vary — the "whisper.cpp
is the clear winner" conclusion in docs/android-asr-runtime-bench.md §9 could be premature for
half the target user base.
Scope
Once Polish eval audio exists (tracked by #63/#79/#30):
- Re-run the same on-device benchmark harnesses already built in
bench/android/{whispercpp,sherpa-onnx}
(Vosk and wav2vec2 don't need re-testing — both were already disqualified on English for
reasons — closed vocabulary gap, no digit tokens — that would only be worse for Polish) against
the Polish clips.
- Compare against whatever Polish desktop CPU baseline exists at that point.
- Land findings as a new section in
docs/android-asr-runtime-bench.md (or a follow-up doc if
it's a large enough addition), per this repo's spike-findings convention — not just a closing
comment.
Non-goals
Blocked by
[android] Validate on-device ASR runtime bench (#120) on Polish audio
Background
Issue #120's Android on-device ASR runtime spike (findings:
docs/android-asr-runtime-bench.md,PR #126) measured all four candidates — whisper.cpp, sherpa-onnx, Vosk, wav2vec2 — on a real
Pixel 7a, but every run so far is English-only. No Polish eval audio exists yet (blocked on
issues #63/#79/#30), so whisper.cpp's 89% E2E and sherpa-onnx's 87% E2E on-device numbers are not
yet confirmed to hold for Polish queries, even though both are advertised as multilingual models.
Why this matters
This project's stated goal is a bilingual (PL+EN) physicist assistant. The whole android runtime
decision in #120 was made on English-only evidence. If Polish accuracy is meaningfully worse
on-device — a real risk, since Whisper's per-language accuracy is known to vary — the "whisper.cpp
is the clear winner" conclusion in
docs/android-asr-runtime-bench.md§9 could be premature forhalf the target user base.
Scope
Once Polish eval audio exists (tracked by #63/#79/#30):
bench/android/{whispercpp,sherpa-onnx}(Vosk and wav2vec2 don't need re-testing — both were already disqualified on English for
reasons — closed vocabulary gap, no digit tokens — that would only be worse for Polish) against
the Polish clips.
docs/android-asr-runtime-bench.md(or a follow-up doc ifit's a large enough addition), per this repo's spike-findings convention — not just a closing
comment.
Non-goals
re-open runtime selection.
Blocked by