Skip to content

[android] Validate on-device ASR runtime bench (#120) on Polish audio #127

Description

@grzanka

[android] Validate on-device ASR runtime bench (#120) on Polish audio

Background

Issue #120's Android on-device ASR runtime spike (findings: docs/android-asr-runtime-bench.md,
PR #126) measured all four candidates — whisper.cpp, sherpa-onnx, Vosk, wav2vec2 — on a real
Pixel 7a, but every run so far is English-only. No Polish eval audio exists yet (blocked on
issues #63/#79/#30), so whisper.cpp's 89% E2E and sherpa-onnx's 87% E2E on-device numbers are not
yet confirmed to hold for Polish queries, even though both are advertised as multilingual models.

Why this matters

This project's stated goal is a bilingual (PL+EN) physicist assistant. The whole android runtime
decision in #120 was made on English-only evidence. If Polish accuracy is meaningfully worse
on-device — a real risk, since Whisper's per-language accuracy is known to vary — the "whisper.cpp
is the clear winner" conclusion in docs/android-asr-runtime-bench.md §9 could be premature for
half the target user base.

Scope

Once Polish eval audio exists (tracked by #63/#79/#30):

  • Re-run the same on-device benchmark harnesses already built in bench/android/{whispercpp,sherpa-onnx}
    (Vosk and wav2vec2 don't need re-testing — both were already disqualified on English for
    reasons — closed vocabulary gap, no digit tokens — that would only be worse for Polish) against
    the Polish clips.
  • Compare against whatever Polish desktop CPU baseline exists at that point.
  • Land findings as a new section in docs/android-asr-runtime-bench.md (or a follow-up doc if
    it's a large enough addition), per this repo's spike-findings convention — not just a closing
    comment.

Non-goals

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions