[android] On-device field-capture: one-tap save of recording + full pipeline trace for adb pull, plus the rotation/lifecycle fixes it depends on

[grzanka]

Summary

The Android full-app (bench/android/full-app) has turned out to be the fastest way to find NLU
and ASR problems — you talk to it, and when the answer is wrong you've discovered a real failing
sentence. Right now that discovery is lost unless someone stops and writes an issue by hand, and
the audio (the actually irreproducible part) is gone the moment the next recording starts.

This proposes an on-device field-capture feature: a one-tap "save this" that writes the WAV plus
a complete pipeline trace to app-private storage, an optional one-line note about what was wrong,
and a PC-side import + labeling pipeline that turns a session of captures into eval corpus rows.

It also covers the orientation/lifecycle bugs that make field testing unpleasant today — which
are not cosmetic: rotating the phone mid-recording currently leaks the microphone, and rotating at
any time reloads the ~639 MB ASR model. A capture feature that loses captures on rotation would be
worse than useless, so these are in scope here rather than split off.

Scoped deliberately with #160 and #159 in mind — see §7.

---

1. Why this is worth building (and why now)

eval/intents.jsonl is a hand-authored, frozen regression suite. #160 measured what that means in
practice: pnpm coverage:intents now reports 100.0% | 100.0% on every single tag, zero examples
below the 0.80 confidence band, and an empty miss list. Every regex fix since #26 arrived together
with the eval rows that cover it, so the set measures "did we remember to add a row for a bug we
already fixed" rather than "does the grammar generalize to phrasings nobody has written down yet."

Field captures are the missing input to that problem. They are, by construction, sentences
nobody sat down and invented — real phrasings, real prosody, real ASR degradation, produced by
someone actually trying to get an answer. That is exactly the held-out, adversarial material #160
argues the project has no source of. #147 is the proof of concept: one real recording ("stopping
power of twenty MeV proton in silicon" → ASR emitted "Me V" as two words) exposed a latent bug in
the web app's en.ts correction rules that the entire hand-authored eval set had never once
triggered. That took a manual bug report, a manual reproduction, and a manual root-cause session.
This feature is the pipeline that makes finding the next one cheap.

Secondary benefits worth stating, because they change the design:

• The raw PCM is the irreplaceable artifact. Transcripts can be regenerated by any future model;
  the audio cannot. Capturing it means a new ASR candidate can be re-scored over the whole field
  corpus offline, with no re-recording session (the same leverage eval/audio/'s 289 existing WAVs
  already provide for the prompted sets).
• Free cross-runtime agreement data. A capture holds the Kotlin matcher's output for a real
  sentence. Replaying that transcript through the TypeScript matcher on the PC yields a per-capture
  agreement datapoint, extending KotlinMatcherAgreementTest from 83 synthetic sentences
  (docs/android-full-app-spike.md §4) to real field speech — where the drift actually matters.

---

2. Current behavior on rotation (measured from the code)

All of the following are in bench/android/full-app/app/src/main/java/com/aidedx/fullapp/MainActivity.kt
and app/src/main/AndroidManifest.xml on current main.

MainActivity extends plain android.app.Activity, the manifest declares no
android:configChanges, and there is no onSaveInstanceState override. So a rotation is a full
destroy → recreate, and all state lives in Activity fields. Consequences, in rough order of severity:

(a) Rotating while recording leaks the microphone — a real bug, not a glitch.

override fun onDestroy() {
    super.onDestroy()
    autoStopHandler.removeCallbacks(autoStopRunnable)
    transcriber?.release()
}

recorder is never touched. AudioRecorder's reader thread loops while (recording) and only
stop() sets recording = false, joins the thread, and releases the AudioRecord. Rotate
mid-recording and: the flag stays true forever, the reader thread never exits, the AudioRecord
is never released (the mic stays hot), and recordedSamples grows unbounded for the life of the
process. The recreated Activity meanwhile sees recorder == null and offers a fresh "Tap to record".

(b) Every rotation releases and reloads the ASR model. onDestroy() calls
transcriber?.release(); onResume() → refreshState() → if (transcriber == null) loadTranscriberInBackground(entry), and ParakeetTranscriber loads its model files from disk in its
init {} block. That is a multi-second reload of a ~639 MB model set on every single rotation —
almost certainly the "app restarts" behavior described.

(c) The 15 s auto-stop cap silently disappears. autoStopHandler.removeCallbacks() in
onDestroy() cancels it, and nothing re-arms it. Combined with (a), a recording started before a
rotation now runs forever with neither a Stop button wired to it nor its safety cap — the exact
multi-minute silent-empty-transcript failure mode #143 added the cap to close.

(d) Rotating mid-transcription discards the result and leaks the Activity.
processRecordingInBackground() starts a bare Thread that calls runOnUiThread { … } on the
Activity that started it. After recreation that Activity is destroyed; the transcript, intent, and
answer are delivered to dead views and vanish with no error.

(e) All displayed results are cleared. No onSaveInstanceState, so transcript/intent/result
text reset to empty — the thing you rotated to read better is the thing rotating destroys.

(f) No landscape layout. There is no res/layout-land/; main.xml is a single vertical
LinearLayout inside a ScrollView, so landscape just squeezes the same column.

---

3. Proposed UI

Design constraints taken from #144's reskin, which established that the record button is the
primary action and that secondary/infrequent actions belong in the toolbar overflow (that's why
"Manage downloads" moved there). Capture must not compete with the mic button, and must never
interrupt the flow of asking the next question.

3.1 Primary affordance: a contextual action on the result

After each query completes, a compact action row appears beneath the answer:

┌──────────────────────────────────────────────┐
│ Transcript:  what is the range of twenty     │
│              Me V proton in silicon          │
│ Intent:      No match                        │
│ Answer:      —                               │
│                                              │
│  ⚑ Save capture        ⌄ details             │
└──────────────────────────────────────────────┘

One tap saves everything. Rationale: the moment you know something is wrong is the moment you're
looking at a wrong answer — no mode to enter first, no setup, no decision about what to include.

Tapping expands a small, optional detail sheet rather than blocking on it:

┌──────────────────────────────────────────────┐
│  What went wrong?  (optional)                │
│   [ ASR ] [ Intent ] [ Number ] [ Slow ]     │
│   [ Other ]                                  │
│                                              │
│  What did you actually say / expect?         │
│  ┌────────────────────────────────────────┐  │
│  │                                        │  │
│  └────────────────────────────────────────┘  │
│                                              │
│              [ Skip ]   [ Save ]             │
└──────────────────────────────────────────────┘

The verdict chips and the free-text line are the highest-value fields in the whole capture and cost
one tap: they are what later turns an unlabeled WAV into a labeled eval row. You are the domain
expert holding the phone — "I said silicon, it heard silicone" written at capture time is worth far
more than a transcript archaeologist reconstructing it a week later. Both are optional; "Skip" saves
the capture unannotated so nothing is ever lost to a dialog.

Confirmation is a non-blocking toast with Undo ("Capture saved · 12 total · Undo"). Never a modal.

3.2 Session mode: "capture everything"

A toggle for a deliberate field-testing session, so you don't have to tap after every utterance:

• lives in the capture screen (§3.3), surfaced as a small persistent indicator in the toolbar when on
  (⚑ 12), because silently recording everything must be visible;
• when on, every query is saved automatically, and the result row's action becomes "⚑ Flag this
  one" — which just sets the verdict/note on the already-saved capture.

This is the mode that matters for the described workflow: talk to the app for twenty minutes, flag
the handful that misbehaved, pull the lot afterwards.

3.3 Capture manager (toolbar overflow → "Debug captures")

A second Activity, alongside ModelManagerActivity and following its conventions:

┌──────────────────────────────────────────────┐
│  Debug captures                              │
│  18 captures · 24.3 MB · session "lgpixel"   │
│                                              │
│  [ Capture everything      ●━━ ON  ]         │
│  Session tag:  [ lgpixel            ]        │
│                                              │
│  ── 2026-07-30 ──────────────────────────    │
│  ⚑ 14:22:07   "range of twenty Me V…"        │
│      ASR · No match · 3.1 s                  │
│    14:21:40   "stopping power of a 150…"     │
│      ok · stoppingPower · 2.8 s              │
│  ⚑ 14:19:02   "how far does a carbon…"       │
│      Intent · csdaRange · 3.4 s              │
│  …                                           │
│                                              │
│  [ Export to Downloads ]   [ Delete all ]    │
└──────────────────────────────────────────────┘

• Session tag names the output directory, mirroring DataGenActivity's speaker extra so
  import-field-captures.sh can take the same argument shape as import-datagen-session.sh.
• Tapping a row opens the full trace (read-only, scrollable) and allows editing the verdict/note
  after the fact, plus play-back of the WAV — being able to hear what you said, on the phone,
  right after seeing the transcript, is often enough to diagnose without any PC involvement.
• Export to Downloads copies the session to shared storage so it can be pulled from a
  non-debuggable build or moved without adb (see §5's run-as caveat).
• Delete all is prominent and honest about size. Storage matters: 16 kHz mono 16-bit PCM is
  ~32 KB/s, so ~2 MB per minute of speech.

3.4 Privacy and honesty

Captures contain recordings of the user's voice and stay in app-private storage. Nothing is ever
uploaded, and there must be no code path that could — the transport is adb, deliberately. The
capture screen states this in one line, shows the total size, and offers deletion. A visible
indicator whenever "capture everything" is on is a requirement, not a nicety.

---

4. What a capture contains

This is the part that determines whether a capture is worth having. The principle: record
everything needed to replay the pipeline offline without the phone, plus everything needed to
explain a discrepancy if the replay disagrees.

Layout under filesDir/captures/<session-tag>/, following DataGenActivity's proven conventions
(per-take WAV + a JSON rewritten after every take so a killed app loses nothing):

filesDir/captures/<session>/
  manifest.json              schema version, session tag, device+app block (written once)
  captures.json              array of capture envelopes, rewritten after each capture
  <captureId>.wav            16 kHz mono PCM — same writeWavFile() as DataGenActivity

Capture envelope (per take)

Identity — captureId (UTC timestamp + short random), wall-clock UTC, local timezone offset,
process uptime, monotonic clock, schemaVersion.

App build — versionName/versionCode, build type, git SHA + dirty flag and build timestamp
injected into BuildConfig by Gradle. Without this a capture from an unknown build is nearly
worthless six weeks later; it's a few lines in build.gradle and should not be skipped.

Device — manufacturer, model, Build.HARDWARE/SoC, Android release + SDK int, ABI, locale,
available/total RAM, low-memory flag, battery %, thermal status. Thermal and battery matter: a
throttled phone gives different latency, and "it got slow" reports are otherwise unfalsifiable.

Audio — sample rate, channels, bit depth, sample count, duration; peak and RMS amplitude,
clipped-sample count, leading/trailing silence duration, rough SNR; mic source; the active
AudioDeviceInfo (built-in mic vs. wired vs. Bluetooth SCO — a Bluetooth headset silently
resampling is a classic cause of "it suddenly got bad"); whether the 15 s auto-stop fired.

ASR — model id, model directory, per-file sizes (and hashes, so a capture can be tied to exact
weights), decoding method, thread count, hotwords file if any; the raw transcript; transcription
wall-clock ms and real-time factor.

Correction — the transcript before and after AsrCorrections, and the list of rule ids that
fired. This is the single most valuable new field for #160's work on the regex layer: today there is
no way to know which of the ~75 correction rules are actually earning their place on real speech, or
which one silently mangled an input. It requires giving the rules stable ids, which is worth doing
anyway.

NLU — the full MatchedIntent serialized as a generic nested object, not a flat set of named
columns (see §7.1), plus which detector path fired (direct keyword / indirect idiom / fuzzy /
default), and an explicit null with a reason when nothing matched. Distinguish "empty transcript"
from "transcript but no match", the same distinction #143 already added to the on-screen line.

Compute — energy converted to MeV/nucl, resolved program, material density, the raw libdedx
outputs (stopping power in MeV·cm²/g, CSDA range in g/cm²), the final formatted answer string, and
per-stage timings.

Failure — any exception type, message, and stack trace, at whichever stage it occurred. A capture
of a crash-adjacent query is at least as valuable as one of a wrong answer.

User annotation — verdict chips, free-text note, and whether the capture was manual or automatic
(session mode). Automatic captures are unbiased samples; manual ones are a biased sample of failures,
and the analysis must be able to tell them apart.

---

5. PC-side pipeline

Two scripts, both modeled on what already exists.

scripts/import-field-captures.sh — near-clone of import-datagen-session.sh, including the
adb shell run-as <pkg> sh -c 'cd files/captures/<session> && tar cf - .' streamed into a local
tar xf - trick that repo already uses to read app-private storage without root. Same
--from-dir escape hatch so it can be exercised with no phone attached. Lands as:

• eval/audio/field-<session>-<date>/ — WAVs, gitignored, same convention as every other session;
• eval/results/field-<session>-<date>/ — captures.json + manifest.json, git-tracked.

Caveat to document: run-as only works on debuggable builds. That's fine for a debug tool, and
§3.3's "Export to Downloads" is the fallback.

scripts/label-field-captures.ts — the genuinely new piece. Unlike a DataGenActivity session,
field captures have no ground truth: the user spoke freely rather than reading a prompt, which is
the entire point. Something has to supply the gold label. A small review tool — play the WAV, show
the transcript/intent/answer and the on-phone note, then enter the correct QueryIntent — is the
step that converts a capture into an eval row. It should pre-fill from the matcher's own output so
the common case is "confirm", not "type".

Output goes to a new eval/field-intents.jsonl, not into eval/intents.jsonl. This is
deliberate and matters: intents.jsonl is the frozen regression suite, and #160's whole argument is
that folding every newly-found sentence back into it is what saturated it to 100%. Field captures
should be the held-out generalization set that #160 says the project has no source of. Same
schema, separate file, separate reported number, plus a "field-capture" tag. Rows can be promoted
into intents.jsonl once a fix lands and they should become regression-protected — but that's an
explicit decision per row, not the default.

Once labeled, the existing scoring already applies unchanged: scripts/asr-score-slots-generic.mjs
and scripts/e2e-audio-intents.ts both consume the established results contract, and the Kotlin ↔
TypeScript agreement check gets a real-speech corpus for the first time.

---

6. Orientation and lifecycle fix

Option A — android:configChanges (stopgap)

One manifest line (orientation|screenSize|screenLayout|keyboardHidden) prevents recreation
entirely, so model, recorder, threads, and displayed results all survive. It preserves the
single-Activity, no-ViewModel convention docs/android-full-app-spike.md explicitly chose ("no
Fragments/Compose/coroutines/ViewModel — matches every other bench/android app's convention"). It
does not give automatic layout-land/ switching, and it's the discouraged modern path.

Option B — retained state via ViewModel (recommended)

androidx.appcompat:appcompat:1.7.0 is already a declared dependency, so androidx.lifecycle is
on the classpath at no new cost — but MainActivity extends plain android.app.Activity, so it must
first become ComponentActivity/AppCompatActivity. Then:

• the loaded ParakeetTranscriber, the active AudioRecorder, the in-flight transcription, and the
  last result move into a ViewModel that survives configuration change — fixing (b), (c), (d) and (e)
  from §2 at the root rather than by suppressing recreation;
• the UI re-renders from observable state on recreate, which also makes a real res/layout-land/
  possible;
• background work stops holding an Activity reference.

This is a deliberate deviation from the stated bench-app convention, and the issue should own
that: the app is graduating from a benchmark spike into a field-testing instrument, and "capture in
progress is lost on rotation" is not an acceptable property for an instrument.

Recommendation

Land A immediately (one line, unblocks field testing today), then B as the real fix — and fix
the microphone leak (§2a) independently of both, since onDestroy() failing to stop the recorder
is a bug regardless of whether recreation is suppressed. Also add, under either option:

• layout-land/main.xml — two-column in landscape (controls left, transcript/intent/answer right,
  independently scrollable) rather than the squeezed single column;
• onSaveInstanceState for the displayed text as belt-and-braces against process death;
• capture writes performed off the Activity's scope entirely (an application-scoped writer), so a
  rotation or a backgrounding mid-write can never truncate captures.json.

---

7. Broader scope: designing for what's coming

7.1 Forward-compatibility with #160 (multi-quantity)

#160 proposes QueryIntent.alsoReport?: Quantity[] and a Kotlin Quantity set. If the capture
envelope serializes the intent as flat named fields (quantity, particleMatch, particleId,
…, mirroring today's MatchedIntent constructor), every one of those changes is a breaking capture
format change and old captures become unreadable.

So: serialize the intent as a generic nested JSON object mirroring the TypeScript QueryIntent
shape (particles: [...], materials: [...], energies: [...], assumptions: [...]), with the
importer preserving unknown fields verbatim. This costs nothing now, survives #160's alsoReport and
#159's MaterialSlot.densityOverride without a schema bump, and has the side benefit that a capture's
intent block can be diffed directly against the TypeScript matcher's output — which is precisely what
the cross-runtime agreement check needs.

schemaVersion on every envelope, and an importer that refuses unknown major versions loudly rather
than silently mis-parsing.

7.2 Runtime-agnostic envelope

The same capture concept applies to the web app — same pipeline, same failure classes, different
runtime. If the envelope is designed now as runtime-agnostic (a runtime: "android" | "web" field
plus per-runtime blocks), one importer and one labeling tool serve both, and field data from the two
becomes directly comparable. Costs one field today; costs a migration later.

7.3 Relationship to existing infrastructure

This should reuse rather than reinvent: writeWavFile() and the rewrite-after-every-take
crash-safety pattern from DataGenActivity, the run-as + tar pull from
import-datagen-session.sh, the eval/audio/ + eval/results/ split, and the existing scoring
scripts' results contract. The one genuinely new component is the labeling step (§5), which exists
because field captures have no prompt to derive ground truth from.

7.4 Non-goals for v1

Automatic upload or crash reporting (transport is adb, deliberately); on-device replay/regression
running; capturing on the web app (designed for, not built); any capture of data beyond what the app
itself produced.

---

8. Risks

• Storage growth. ~2 MB/minute of speech. Needs a size cap with oldest-first pruning, a visible
  total, and easy deletion — especially with "capture everything" on.
• Privacy. Voice recordings, held locally, never uploaded, visibly indicated, easily deleted.
• Capture cost on the hot path. WAV writing must not block the audio reader thread or delay the
  answer; write asynchronously after the answer renders.
• Biased corpus. Manually-flagged captures over-represent failures. Session mode's automatic
  captures are the unbiased baseline; the envelope records which is which, and any reported accuracy
  number must respect the distinction.
• Labeling is human effort. The value of the whole pipeline is gated on someone labeling
  captures. Pre-filling from the matcher and keeping the reviewer's job to "confirm or correct" is
  the difference between this being used and abandoned.

---

9. Checklist

Lifecycle / orientation (independently valuable, land first)

• Fix the microphone leak — onDestroy() must stop the recorder (bug, independent of everything else)
• Stopgap: android:configChanges on MainActivity
• MainActivity → ComponentActivity/AppCompatActivity; retained state in a ViewModel
• Transcriber, recorder, in-flight transcription, and last result survive rotation
• Re-arm the 15 s auto-stop across configuration change
• res/layout-land/main.xml two-column landscape
• onSaveInstanceState for displayed text
• Manual device test matrix: rotate while idle / recording / transcribing / downloading

Capture core

• Capture envelope schema (schemaVersion, runtime-agnostic, generic intent object) — write it down before coding
• Git SHA + build timestamp into BuildConfig via Gradle
• Stable rule ids in AsrCorrections + record which fired
• Application-scoped capture writer (WAV + captures.json rewrite-after-each, crash-safe)
• Audio quality metrics (peak/RMS/clipping/silence, routed device)
• Per-stage timings + exception capture

UI

• "Save capture" action on the result row + verdict chips + optional note sheet
• Non-blocking toast with Undo
• "Capture everything" session mode + visible on-state indicator + session tag
• Debug captures Activity: list, detail view, WAV playback, export to Downloads, delete all
• Privacy line + total size shown

PC pipeline

• scripts/import-field-captures.sh (run-as + tar, --from-dir)
• scripts/label-field-captures.ts — review + gold-label tool, pre-filled from the matcher
• eval/field-intents.jsonl (new held-out corpus) + "field-capture" tag
• Replay captured transcripts through the TS matcher → real-speech Kotlin↔TS agreement number
• Docs: capture format reference + the field-session workflow, per CLAUDE.md's findings-doc rule