[android] Part 5 of #130: synthesize the 50+50 datagen sentences on Athena TTS for a real human-vs-synthetic comparison

[grzanka]

Background

#130's own Part 5 (explicitly deferred, not in PR #131): "The display strings are usable as TTS
input directly — part of why units are written as read. Synthesizing the same 100 sentences (Qwen
/ Chatterbox PL, existing scripts) gives a human vs. synthetic comparison on identical text, which
no prior batch in this repo has: the 1000-sentence sets have no human counterpart, and
eval/audio/{km,lg,mn} has no synthetic one. Filed as its own issue once Parts 1–4 land."

Parts 1–4 have now landed (#131, and #134 closed via #149/#150), and #130 itself is closed. We now
have the human half of that comparison for real: #149 recorded a full 100/100 (50×EN + 50×PL)
session on a real Pixel 7a (speaker lgpixel) of exactly the eval/datagen-sentences.json set,
scored in docs/android-datagen-bench.md. This issue is the deferred Part 5: synthesize the
same 100 display strings via TTS on Athena, transcribe with the same pipelines already used
for lgpixel, and score with the existing scripts/e2e-audio-intents-datagen.ts harness — a
genuine same-text, same-pipeline, human-vs-synthetic table this repo has never had.

The "spell it like humans" question — already substantially answered

eval/datagen-sentences.json's display field (distinct from canonical) was purpose-built for
TTS input from the start — #130 Part 1 §1.2 and eval/RECORDING.datagen.md already encode
human-pronunciation-informed spelling for every acronym/unit that appears in this set (confirmed
by grepping the committed JSON: CSDA, LET, PMMA are the only acronyms anywhere in the 50
tuples, so there's no larger unaddressed vocabulary to cover):

• Length units (cm/mm/um) always spelled out — "10 centimeters".
• Energy units (keV/MeV/GeV) — bare abbreviation in 45/50 (real speakers read these as
  compact acronyms, "mef"/"kef"/"Gef", no hint needed or wanted — docs/unit-pronunciation-asr.md
  §5.1), spelled out in the other 5/50 ("kiloelectronvolts").
• LET — letter-spelled in English display ("el-ee-tee"), left as the one-syllable
  borrowed word in Polish.
• CSDA — letter-spelled in English display ("see-ess-dee-ay"), unchanged in Polish
  (matches eval/RECORDING.pl.md's own precedent for how it's actually said).
• PMMA — deliberately given no rendering rule in either language: reasoned (not
  measured) to already be unambiguous letter-by-letter, the same category as the abbreviated
  energy-unit majority.

What's not yet done: actually verifying any of this against a real TTS engine's output. Every
rule above is reasoned from human-speech convention or copied from precedent, not measured against
what the chosen engine's G2P actually produces for this set. PMMA is the one case flagged
above as an assumption rather than a finding — worth an explicit check of whether it comes out
sounding wrong on any candidate engine, not just carried forward unverified.

Which engine(s) — best per language, on Athena

State of the art as measured in this repo, most-recent-doc wins (check before starting — this is
a moving target):

• English: Qwen3-TTS (VoiceDesign preset pool) — the established engine across
  docs/tts-eval-1000.md → -v2 → -v3, 84.0% clip-level (new-corrector) on the
  abbreviation-only v3 corpus. Existing script: scripts/tts-qwen-1000.py.
• Polish: less settled. docs/tts-eval-1000-pl.md found Piper the only currently-viable
  engine (20.8% vs. Qwen3-TTS PL's ~0.9% — Qwen3-TTS literally rejects language="Polish", not
  just lower quality). Chatterbox multilingual (native + voice-cloned,
  docs/tts-chatterbox-pl-clone.md) was built specifically to close that gap but has no results
  doc yet — an in-progress Athena session is still working through its real 0.1.7 API surface
  as of this writing. Use whichever engine has the most recent completed result when this issue
  is picked up, rather than assuming Piper.
• Since this is only 100 clips (not 1000), it's cheap to run more than one candidate per language
  rather than committing to a single "best" pick up front — e.g. both Qwen3-TTS and a second
  English engine, and however many Polish engines are viable at the time — so the human-vs-synthetic
  comparison isn't accidentally confounded by one engine's specific weaknesses. Nice-to-have, not
  the core ask if it meaningfully grows scope.

Deliverables

• A small generation script (same shape as scripts/tts-qwen-1000.py / tts-piper-1000.py /
  tts-chatterbox-1000-pl.py, much shorter at 100 clips instead of 1000) that reads
  eval/datagen-sentences.json's display field per id/language, synthesizes on Athena (sbatch,
  same pattern as scripts/submit-v3.sh / scripts/submit-pl.sh), and writes into the same
  {modelId, dtype, withPrompt, loadS, records[]} results contract
  scripts/e2e-audio-intents-datagen.ts already consumes for the real lgpixel session — so
  scoring is free, not new code.
• Transcribe the synthesized audio with the same pipeline(s) already scored for lgpixel (desktop
  whisper-small+prompt at minimum, since that runs on Athena directly; on-device
  Parakeet/Whisper/whisper.cpp would need the phone rather than Athena, optional).
• Score with scripts/e2e-audio-intents-datagen.ts, same as docs/android-datagen-bench.md §4.1
  already does for lgpixel.
• Findings land in docs/android-datagen-bench.md (a new section) or a new
  docs/android-datagen-tts-bench.md if it gets long — per this repo's spike-findings convention
  — with at minimum:
  • A head-to-head table: lgpixel (real) vs. synthetic, same 100 sentences, same scoring
    pipeline.
  • Whether the display field's pronunciation hints (expanded-energy 5/50, LET/CSDA
    letter-spelling, PMMA left alone) actually produce audio that resolves correctly on the
    chosen engine(s) — a per-acronym/per-rendering breakdown, not just one aggregate number.
  • An explicit call-out if PMMA's "no rendering rule" assumption doesn't hold on some engine.

Non-goals

• Not re-deriving pronunciation rules for units/acronyms from scratch —
  eval/RECORDING.datagen.md already made those calls; this issue verifies them against real TTS
  output, it doesn't redesign them.
• Not the 1000-sentence-batch TTS track (issue [i18n] Define sentence/speaker counts for the Polish eval set (recorded vs. generated) #79/[eval] TTS-1000 v2 methodology follow-ups: per-unit/stratified reporting, DOMAIN_PROMPT expansion, voice-composition fixes #92 family) — different corpus, different
  purpose (broad coverage vs. this issue's identical-text human/synthetic pairing).
• Not voice cloning of the lgpixel speaker. Issue [eval] Explore Chatterbox TTS for Polish voice cloning / native synthesis (issue #79 follow-up) #106's consent precedent (cloning requires
  the recorded person's explicit authorization) would need to be re-confirmed for this speaker
  before that variant is in scope. Native (uncloned) synthesis needs no such step and covers the
  core comparison.

Related

• [android] Benchmark data generator app — prompt, record, dual-model transcribe, export a bilingual dataset #130 — parent issue; this is the deferred Part 5
• [issue-130] Part 4: run the real datagen recording session, score it, land findings doc #134 / [issue-134] Land real DataGenActivity recording session (lgpixel, 100/100 prompts) #149 / [issue-134] Cross-runtime comparison + findings doc (desktop whisper-small) #150 — the real lgpixel recording session + cross-runtime findings this issue
  compares against
• Energy-unit ASR correction: TeV support + a silent TeV→MeV miscorrection bug, plus a 0%-coverage gap on spoken-out unit readings #151 / Energy-unit corrector: NUMBER_PREFIX_SRC doesn't handle spelled-out hundreds ("five hundred kilo electronvolt" still fails) #153 — energy-unit corrector fixes whose real-audio evidence came from this same 50+50
  set
• [i18n] Define sentence/speaker counts for the Polish eval set (recorded vs. generated) #79 / [eval] TTS-1000 v2 methodology follow-ups: per-unit/stratified reporting, DOMAIN_PROMPT expansion, voice-composition fixes #92 / [eval] Explore Chatterbox TTS for Polish voice cloning / native synthesis (issue #79 follow-up) #106 / Unit pronunciation in eval audio vs. ASR transcripts — investigation & TODO #118 — prior TTS-engine and pronunciation research this issue builds on
  (docs/tts-eval-1000*.md, docs/tts-chatterbox-pl-clone.md, docs/unit-pronunciation-asr.md)