Skip to content

[audit] Bug hunt + architecture/maintainability review: 22 bugs (B1-B7 fixed; 2 regressions + 10 new found on re-audit), cross-layer contract drift as the root pattern, libdedx forward-compat, and a docs/benchmark-corpus reorganization plan #163

Open
Open
[audit] Bug hunt + architecture/maintainability review: 22 bugs (B1-B7 fixed; 2 regressions + 10 new found on re-audit), cross-layer contract drift as the root pattern, libdedx forward-compat, and a docs/benchmark-corpus reorganization plan#163
Labels
bugSomething isn't workingenhancementNew feature or requesteval

Description

@grzanka
grzanka
opened on Jul 30, 2026

Summary

A full-repo bug hunt and engineering audit against main (397e1df), plus a concrete plan for the
docs/ + benchmark-corpus reorganization. Three parts:

  1. §2–§3 — ten bugs, all reproduced against the real vendored WASM and quoted with measured
    output. Four of them show a confidently wrong physics number to the user with no error, no
    warning, and no plausibility flag. Two are wrong by ~200× and ~18×.
  2. §4–§6 — architecture, maintainability, and engineering practice. Including one pattern that
    explains four of the ten bugs and has a cheap, type-level fix.
  3. §7–§8 — docs/ and the benchmark corpus. With the measurement that matters: not one
    benchmark number in this repo is reproducible by a second person    , because /eval/audio/ is
    gitignored. That is the root cause of the docs mess, and it means the reorganization has to start
    with the corpus, not the docs.

Every CI gate is green — format:check, lint, check (949 files, 0 errors), validate:eval,
pnpm test (808 tests / 42 files, all passing). Every bug below is invisible to all of them. That
is the finding, not an aside.

Reproduction harness for §2: matchIntent() → computeIntent() → renderAnswer() driven against
static/wasm/libdedx.mjs loaded in Node, exactly as compute.smoke.test.ts does. Every "measured"
line is verbatim output.

🔄 Re-audit status — 2026-07-31 (HEAD 8410868)

B1–B7 are verified fixed against the real WASM (#166, #167), B8/B9/B10 remain open, and
#160 §8's bench:nlu has landed (#168) and is wired into CI. A second bug hunt over the same
ground found 12 further bugs — 2 of them regressions introduced by the B3/B5/B6 fixes
themselves, 2 more silently-wrong-number P1s, and one incomplete fix (B3 is English-only). Every
one is invisible to all 5 CI gates, which are green (900 tests, 954 files typechecked).

Full measured re-audit, including a libdedx forward-compatibility assessment for
electron (ESTAR) and projected-range support, is in
this comment. The findings are folded into §10 below as C1–C12.
Sections §1–§9 are left as the original record of the audit against 397e1df; the measurements
in them are historical, not current (§7's have since got worse — see the comment).

✅ Status update — 2026-08-12

All of C1–C4, C6, C7, C9–C12, B8, B9, B10 are now fixed (#176, #177, #178), closing every
"silent wrong number" and every "misleading answer" item in §10 except C8 (Polish i18n for
the unresolved-detection regexes — still open). The instrument batch's contracts.test.ts
and the libdedx forward-compat batch's QUANTITY_KIND/isInverseQuantity() also landed
(#178). The three process items (§6.1 gate-deploy-on-CI, §6.2 dependabot coverage, §6.3
Gradle test job) landed in #179, and dependabot's github-actions/gradle coverage from §6.2
has since been exercised end-to-end: it opened 20 real update PRs, all triaged and merged
(#180–#201), plus one follow-up (#200) that groups kotlin-stdlib+kotlin-gradle-plugin per app
(the split was itself the cause of one of those PRs failing CI) and matrices Android Bench Tests across all five bench/android/* apps instead of only full-app.

Checkboxes below are updated to match. Still open: C8; the three remaining instrument
items (mutation-layer generators, a programs gold field, lang: "pl" eval runs); three of the
four libdedx forward-compat items (ComputePoint keying, electron/ESTAR, projected range);
§5.3/§5.4 (COI service worker, offline-claim verification); and all of §8 (corpus + docs
reorg) — none of these have any code or doc changes on main yet, verified directly against the
current tree rather than inferred from PR titles.

✅ Status update — 2026-08-13

§5.3 landed (#211 — promoted the COI service worker, dropped the stale DEBUG comment;
doc-only, no behavior change). C8 and two of the three instrument items landed together
in #214: unresolved-material/particle detection is now LangPack-driven (Polish gets the same
honest "not specified" banner English does, instead of silently substituting a default), a
programs gold field now exists in coverage.ts's FieldVerdicts/compareIntent(), and
bench:nlu gained a round-hundreds (C5b) and an "in <unit>" (C4) regression generator. #214's
Copilot review also caught a real Unicode-\b truncation bug in the new Polish regex, fixed in
the same PR and verified against the real matcher.

Checkboxes below are updated to match. Still open: running the eval/bench suites for
lang: "pl" (the third instrument item — needs lang threaded through
runCoverage()/buildCanonicalCorpus(), bench-nlu.ts's own header still scopes it to English
v1); three of the four libdedx forward-compat items (ComputePoint keying, electron/ESTAR,
projected range); §5.4 (offline-claim verification); and all of §8 (corpus + docs
reorg) — verified directly against the current tree.

✅ Status update — 2026-08-13 (later)

§5.5 landed (#215) — ComputePoint now carries a single values: Partial<Record<Quantity, number>> keyed directly by Quantity instead of separate stoppingPower/csdaRange/energy
fields, so valueText() (and the same fragile field-name pick, found a second time in
scripts/tts-sentence-check.ts and fixed there too) reads point.values[quantity] instead of a
ternary that would silently mis-render a third forward quantity.

§8 Phase 5 landed (#216) — docs/coverage.md, generated from the alias tables plus the real
libdedx WASM's own getParticles()/getMaterials() per program, listing every particle/material
phrase the app recognizes and a computed (not hand-written) "known gaps" section — currently the
electron (ESTAR is a listed program with zero tabulated materials) and elements Z=113-118, which
no program lists at all. A coverage-doc.smoke.test.ts freshness guard mirrors aliases.test.ts's
JSON-artifact check.

Checkboxes below are updated to match. Still open: running the eval/bench suites for
lang: "pl" (instrument); two of the three remaining libdedx forward-compat items
(electron/ESTAR, projected range); §5.4 (offline-claim verification); and §8's corpus +
docs reorg core, Phase 0-4 (Phase 5 alone is done) — verified directly against the current tree.

✅ Status update — 2026-08-13 (§5.4 verified)

§5.4 is now verified, not just speculated: live browser testing (Firefox on Linux, against
the deployed https://aptg.github.io/aidedx/) found the offline claim fails on first use —
wasm/libdedx.mjs and the ASR worker script are both lazily loaded on first query and never
touched by the "download models" flow, so neither is cached until a query actually runs. It
succeeds once one query has completed online in the same tab session. The originally
suspected culprit (ORT wasm from jsdelivr) was never even reached — the app fails earlier, on its
own same-origin code. Full repro, root cause, and a two-part fix (precache the compute/ASR paths
during download; give coi-serviceworker.js's fetch handler a caches.match() fallback) are in
#217. Checkbox below reflects that the verification is done; the fix itself is not — tracked in
#217, still open.

1. What's working — stated first, because it changes how to read the rest

This is a well-disciplined repo, and several things are genuinely better than typical:

  • CI gate is real and enforced: 5 gates, documented in CLAUDE.md in the order CI runs them,
    plus two non-blocking reported metrics. 808 tests, 42 files. tsc runs with
    exactOptionalPropertyTypes and the code respects it (withDensity() in compute.ts:137).
  • The "numbers never come from the LLM" invariant holds. I looked for a hole and did not find
    one: every value in every answer traces to a libdedx call.
  • Rationale is written down. 28% of non-test lines in src/lib are comments, and they carry
    why, with issue numbers — atomicMassForConversion()'s docblock (compute.ts:358-372) explains
    a real physics subtlety and the bug it fixed. This is the right instinct.
  • Layering is clean where it counts: src/lib/wasm/ has no QueryIntent dependency and is
    genuinely extractable; loader.ts has no SvelteKit import, with sveltekit.ts as the adapter.
  • Cross-runtime discipline exists: KotlinMatcherAgreementTest is the right idea for keeping two
    matchers honest (§6.3 is about the fact that nothing runs it).

So the problem is not care or rigor. It is that the gates measure the layers, and the bugs live
between them (§4.1).

2. P1 bugs — a wrong number, shown confidently

B1 — micrometre range targets are computed as centimetres (~200× error, silent)

extractRangeTarget() accepts µm and normalizes it to "um":

// src/lib/intent/matcher.ts:475
const LENGTH_TARGET_RE =
  /(\d+(?:\.\d+)?)[\s-]*(g\s*\/\s*cm…|mm|millimeters?|cm|centimeters?|[uµ]m|micrometers?|micron[s]?)\b/i;
// :490
else if (raw === "micron" || raw === "microns" || raw === "µm" || raw.startsWith("micrometer"))
  unit = "um";

rangeTargetToGcm2() has no "um" branch, and its fallthrough is centimetres:

// src/lib/compute/compute.ts:331-335
let cm: number;
if (unit === "mm") cm = target.value / 10;
else if (unit === "m") cm = target.value * 100;
else cm = target.value; // "cm" (default)

Measured:

Query aidedx answers Correct (same program, same WASM) Error
"What proton energy gives a 500 µm range in water?" 1374 MeV 6.012 MeV 229×
"What energy proton has a range of 300 micrometers in water?" 943.3 MeV 4.485 MeV 210×
(control) "…stops after 100 mm in water?" 115.7 MeV 115.7 MeV ✓

confidence: 0.9, incomplete: false, series.error unset, validateIntent() → plausible: true.
Nothing anywhere signals a problem.

Note the shape of this: m (metres, which nobody says) is handled; µm (which the matcher was
deliberately extended to accept in #122, because Parakeet spells length units out) is not. The
producer grew a unit and the consumer didn't.

Also cosmetic but in the same line of output: the answer echoes 500 um, not 500 µm.

B2 — keV/µm stopping-power targets are computed as MeV·cm²/g (18× in water, silent)

STP_TARGET_RES (matcher.ts:502) emits unit: "keV/um", and eval/README.md explicitly documents
keV/um as a valid energyFromStp target unit. stpTargetToMassUnits() handles two units and then
gives up quietly:

// src/lib/compute/compute.ts:348-355
if (unit === "mev·cm²/g" || unit === "mevcm2/g" || unit === "mevcm²/g") return target.value;
if (unit === "mev/cm") { …/density… }
return target.value; // assume mass stopping power if unitless/unknown

Measured — the same physical stopping power, entered two ways:

Query aidedx answers
"What energy proton has a stopping power of 8 keV/µm in water?" 88.28 MeV
"What energy proton has a stopping power of 80 MeV/cm in water?" 4.925 MeV ✓

8 keV/µm = 80 MeV/cm = 80 MeV·cm²/g in water. The correct answer is 4.925 MeV; the app reports
88.28 MeV. In a non-unit-density material the factor changes, so it can't even be spotted as a
constant.

Sharpest detail: the forward direction renders stopping power in keV/µm (render.ts:105,
stoppingPowerToKevPerUm). The app answers in a unit it cannot read back — and #160 §3.2 uses
"8 keV/µm" as an example query, so this is on the roadmap's own path.

B3 — an unrecognized entity is treated as an absent one and silently replaced

fillMissingSlots() (#10's dead-end recovery) fires on empty slot arrays. But the matcher only emits
a slot for a phrase it can resolve, so "a particle libdedx doesn't have" and "no particle
mentioned" are the same state. Measured, full store path (answer-status.svelte.ts:112-124):

Query Banner shown Answer actually given
"range of muons in water at 100 MeV" "particle not specified → proton" range of a proton
"range of pions in water at 100 MeV" "particle not specified → proton" range of a proton
"stopping power of 100 MeV protons in stainless steel" "material not specified → water" in water
"…protons in unobtanium" "material not specified → water" in water

Two problems, both serious for this audience:

  1. The banner is factually false. The user named a particle/material. Being told they didn't is
    worse than a plain error, because it invites them to accept the substitute.
  2. Stainless steel is an ordinary beamline material and muons/pions are ordinary particles.
    "libdedx has no tabulated data for stainless steel" is useful information a physicist needs;
    "here's water instead" is not.

This is precisely #132's failure class — a plausible-looking wrong answer instead of a loud failure —
reintroduced through #10's defaults feature. Fix: give the matcher a third state
(unresolved: [{ kind, phrase }]), let fillMissingSlots() fire only on genuinely-empty slots, and
name the unknown entity in the message.

B4 — spelled-out 3-digit numbers with a tens+ones remainder collapse to the ones digit

composeHundreds() (matcher.ts:98-118) allows exactly one NUMBER_WORDS remainder token, and
NUMBER_WORDS has no multi-word entries. composeTensOnes() — which would compose "thirty five" —
runs after composeHundreds() (matcher.ts:195), so it never gets the chance:

"two hundred thirty five MeV"
  composeHundreds → "230" (consumes "two hundred thirty")  +  orphan "five"
  spellOutNumbers → "230               5    MeV"
  extractEnergies → picks the number adjacent to the unit → 5 MeV

Measured:

Query Parsed energy Answer Correct
"range of a two hundred thirty five MeV proton in water" 5 MeV 362 µm 34.19 cm
"range of a nine hundred ninety nine MeV proton in water" 9 MeV 1.021 mm 3.25 m
(control) "two hundred thirty MeV" 230 MeV ✓ 32.96 cm ✓

confidence: 0.97, incomplete: false. This is the fourth iteration of the same family (#122 →
#147 → #151 → #153 → #156), and it matters for exactly the reason #122 gave: Parakeet has no ASR
inverse text normalization, so a 235 MeV beam arrives fully spelled out. Fix is small — allow a
\d+ remainder in composeHundreds and run composeTensOnes first — but see §4.2 on why fixing it
one more time by hand is the wrong move.

3. P2 bugs — misleading, but not a silently wrong number

B5 — an explicitly requested program is ignored, and the coverage harness cannot see it

matchIntent() never sets intent.program. detectPrograms() output feeds only
decideCompareDim(). Measured: "Using PSTAR, what is the range of 150 MeV protons in water?" →
computed and labelled ICRU49.

The instrument half is worse. eval/intents.jsonl has gold rows that carry the field:

{"id": "prog-001", "text": "Using PSTAR, what is the range of 150 MeV protons in water?",
 "expected": { …, "program": "PSTAR", … }, "tags": [… "program-specified"]}
{"id": "prog-002", … "program": "ASTAR" …}

and FieldVerdicts / compareIntent() (coverage.ts:28-36, 113-123) have no program key.
So both rows score as exact matches, and coverage:intents' headline —

slots  100.0% (122/122)   exact  100.0% (122/122)

— is computed over 6 of the 7 gold slots, omitting the one the matcher never implemented. #160
§8 argues the eval set has stopped discriminating; this is stronger than that. The instrument is not
merely saturated, it has a blind spot exactly where a feature is missing, and it reports 100%.

Latent follow-on: once the matcher does set program, resolveProgramId() (compute.ts:263-268)
silently falls back to auto-select for any name absent from PROGRAM_NAME_TO_ID. Same silent-drop
shape as B1/B2.

B6 — program names libdedx doesn't have silently select three unrelated programs

PROGRAM_RE (matcher.ts:700) matches srim|atima|geant4?|fluka|nist, none of which exist in
PROGRAM_NAME_TO_ID. Two such names ⇒ compareDim: "program" ⇒ compareProgramsForParticle()
ignores what was asked and returns a hardcoded triple. Measured:

Q: "Compare SRIM and ATIMA for the range of 100 MeV protons in water."
→ CSDA range of 100 MeV protons in water, by program:
  - PSTAR: 7.721 cm    - ICRU49: 7.721 cm    - Bethe: 7.699 cm

The user asked about two codes aidedx cannot run and got three others, unremarked. The honest answer
is "libdedx doesn't include SRIM or ATIMA; here's what it does have."

B7 — total→per-nucleon is applied to every A>1 particle, but disclosed only for some

compute.ts:319 divides by A whenever perNucleonAssumed !== true. The matcher only annotates
that when isotopeAssumed is set (matcher.ts:852), which excludes named light ions whose isotope
is inherent. Measured:

Query Computed at Note shown?
"20 MeV deuteron" 10 MeV/nucl no
"20 MeV triton" 6.67 MeV/nucl no
"5 MeV alpha particles" 1.25 MeV/nucl no
"20 MeV carbon ion" 1.667 MeV/nucl yes

The reading is defensible (accelerator convention). The silence and the inconsistency are the
bug: two layers disagree about which particles trigger the assumption, and the user is told for one
and not the other. For a tool whose entire trust story is "we show you what we assumed", that's a
core-value failure, not a nit. Same note also prints 1.666667 / 33.333333 — round()'s 1e-6
precision leaking into physicist-facing prose.

B8 — a multi-particle comparison note quotes only the first particle's per-nucleon energy

Q: "Compare the range of carbon and neon ions in water at 400 MeV."
→ - carbon: 3.626 mm (ICRU73)    - neon: 891.4 µm (ICRU73)
  Note: …; 400 MeV taken as total → 33.333333 MeV/nucl.

The neon series was computed at 20 MeV/nucl. One note, two different per-nucleon energies, and
the one printed is wrong for the second row. Fix: derive the note per series in render.ts from
ComputeSeries, not once in the matcher from particles[0].

B9 — two compare dimensions at once is a hard error with a developer-facing message

decideCompareDim() (matcher.ts:712-723) is a single-winner priority chain (energy > material

particle), and computeIntent()'s #132 defensive assert then throws. Measured:

Q: "What is the range of protons in water and PMMA at 100 and 200 MeV?"
→ ComputeError: compareDim "energy" but 2 materials present — only the first would be computed

That string reaches the user verbatim (answer-status.svelte.ts:181). A completely ordinary physics
question produces an internal invariant message. The assert was the right call in #132 — but the
model behind it is what needs fixing, and #160 §6's ComputePlan is exactly that fix
(compareDim scalar → fan-out set). Until then, at minimum: map ComputeError to a user-facing
sentence and say which dimension was dropped.

B10 — the plausibility gate does not cover inverse queries at all

validateIntent() skips the energy check for inverse quantities (validate.ts:230) and has no
target check of any kind — not value, not unit. Both B1 and B2 return plausible: true. Issue
#29 built the gate for forward energies; the inverse path, which is where both P1 unit bugs live, is
ungated. Two cheap additions would have caught both: a closed unit set (§4.1) and a round-trip check
(forward-compute the resolved energy and confirm it reproduces the stated target — #160 §3.2 already
proposes this for a different reason).

4. The pattern behind the bugs, and the fix

4.1 Cross-layer contract drift — one root cause, four bugs

B1, B2, B5 and B7 are all the same failure:

Producer grew a value Consumer never learned about it Test that would have caught it
matcher emits target unit "um" rangeTargetToGcm2() falls through to cm none
matcher emits target unit "keV/um" stpTargetToMassUnits() falls through to mass units none
eval gold carries program matcher never sets it, compareIntent never checks it none
compute divides by A for all A>1 matcher annotates only isotopeAssumed ions none

Each layer is tested against its own expectations, and nothing tests the seam. That is the
signature failure mode of per-issue LLM-assisted development: the change lands in the file the issue
points at, together with tests written from that same file's point of view, and the sibling layer
stays behind. Both sides look correct in isolation, so review and CI both pass.

The single highest-value fix is a type change. TargetSlot.unit is string by design:

// src/lib/intent/query-intent.ts:69-78
/** … `unit` is a free string because the accepted units differ per quantity … */
export interface TargetSlot { value: number; unit: string; }

That documented decision is what makes B1 and B2 possible. Replace it with closed unions —

export const RANGE_UNITS = ["cm", "mm", "m", "um", "g/cm2"] as const;
export const STP_UNITS   = ["MeV cm2/g", "MeV/cm", "keV/um"] as const;
export type TargetSlot =
  | { kind: "range"; value: number; unit: (typeof RANGE_UNITS)[number] }
  | { kind: "stp";   value: number; unit: (typeof STP_UNITS)[number] };

— and give both converters an exhaustive switch with a never default. B1 and B2 become compile
errors, the matcher can no longer invent a unit the converter doesn't handle, validateQueryIntent
gets a real check instead of typeof unit === "string", and eval/README.md's unit table becomes
generated rather than aspirational.

Generalize it as a rule: every value that crosses a module boundary is an as const union with an
exhaustive consumer switch. The codebase already does this well for QUANTITIES, COMPARE_DIMS,
ENERGY_UNITS, and EVAL_TAGS — the bugs are concentrated precisely in the places it doesn't
(TargetSlot.unit, intent.program, PROGRAM_RE's name list vs PROGRAM_NAME_TO_ID's keys).

Cheap companion, for contracts a type can't express: a contracts.test.ts that enumerates each
producer's vocabulary and asserts the consumer handles every member. For PROGRAM_RE that's four
lines and catches B6.

4.2 The eval set has become a fixture of past bugs

#160 §8 makes this argument; I can add the mechanism as a measurement. Every recent fix has the same
shape: a real recording fails → widen one regex → add eval rows for that phrasing. #26 (6 fixes),
#103 (4), #122 (4), #132, #147, #151, #153, #156. The result is 122 rows at 100%/100% with an empty
misses list and zero examples below the 0.80 confidence band — and now also one gold field that
is never compared at all (B5).

B4 is the proof: it is the fifth member of the spelled-out-number family, it was found by hand
here rather than by any instrument, and it will not be the last. pnpm bench:nlu with a mutation
layer (#160 §8) would have produced all five at once. I'd treat #160 §8 as the highest-priority
open item in the repo, above the multi-quantity feature it was written to support.

4.3 Defensive asserts standing in for model fixes

compute.ts:597-606 throws rather than silently mis-answering — correct, and well-commented. But the
underlying model defect (single-valued compareDim) was never fixed, so B9 is a valid query hitting
an invariant message. Suggested convention: when an assert is added because the model is wrong,
open the model-fix issue and cite it in the comment, so the assert has an exit condition instead of
becoming permanent.

5. Architecture & maintainability

5.1 This is a research monorepo wearing a product repo's clothes

Area Lines Under CI? Under ESLint?
src/ (shipped app) 18,306 ✅ 5 gates ✅
scripts/ (research harnesses) 10,637 ❌ only 3 invoked partly (.mjs/.ts yes, .py/.sh no)
bench/ (5 Android apps, JNI/C) 8,116 ❌ none ❌ explicitly ignored
docs/ 9,586 md ❌ —

Research scaffolding (28.3k lines) is 1.5× the shipped app (18.3k), and only the app half is
governed. That's the structural fact behind both the docs problem (§7) and §5.2/§6.3. It isn't
wrong to keep research in-repo — for this project it's a strength — but it needs its own, lighter
governance rather than none.

5.2 bench/ duplication is measured, and ungoverned

Byte-identical files between bench/android/full-app/ and bench/android/sherpa-onnx/:

IDENTICAL  OfflineRecognizer.kt (1509 lines)   IDENTICAL  WaveReader.kt (60)
IDENTICAL  OfflineStream.kt (38)               IDENTICAL  FeatureConfig.kt (11)
IDENTICAL  QnnConfig.kt (7)                    IDENTICAL  HomophoneReplacerConfig.kt (7)

1,632 lines of vendored sherpa-onnx bindings, copied verbatim, with no shared Gradle module and no
build that would notice them diverging. Plus 4 committed gradle-wrapper.jar binaries.

The NLU duplication #160 §9 describes is real and I can add a number: the stopping-power synonym
concept appears in 19 files across src/, scripts/, and bench/ (tests included). #160's (a)
— lexicons as data files + codegen for TS and Kotlin, following scripts/generate-aliases.ts — is
the right call and I'd raise its priority: it's the only item that shrinks §5.2 and §6.3 at once.

5.3 A DEBUG experiment from a closed issue is shipping to production

src/app.html contains, three lines apart:

<!-- … the planned workaround is `coi-serviceworker`: drop coi-serviceworker.js into static/ and
     register it here, e.g.: <script src="…/coi-serviceworker.js"></script>
     Left intentionally inert until the hosting decision is made. -->

<!-- DEBUG (#9 threading experiment — revertable): … Remove this line + static/coi-serviceworker.js
     to revert. -->
<script src="%sveltekit.assets%/coi-serviceworker.js"></script>

The first comment asserts the thing is inert; the second registers it. Issue #9 closed 2026-07-15.
A vendored third-party service worker that rewrites COOP/COEP for every response, labelled DEBUG,
has been in production for two weeks. Either promote it (delete the stale comment, drop the DEBUG
label, record the decision as an ADR — docs/threading-coop-coep.md has the evidence) or revert it.

This is also the clearest example of comment rot, which matters more here than usual: at 28% comment
density, comments are load-bearing, and nothing checks them.

5.4 Verify the offline claim

README.md promises "no internet needed once it's set up… keeps working inside a heavily shielded
irradiation room". The repo's own comment in static/coi-serviceworker.js lists
"jsdelivr ORT wasm" among the cross-origin subresources, and no wasmPaths override appears
anywhere in src/ — so onnxruntime-web resolves its .wasm from its default CDN. If those bytes
aren't in the Cache Storage the download flow populates, the offline claim fails on a cold cache.

I could not test this in a browser here, so it's a verification item, not a finding — but it's
the app's headline promise and its named use case, so it deserves an explicit test: download models,
go offline, hard-reload, transcribe. If it fails, self-host the ORT wasm via
env.backends.onnx.wasm.wasmPaths (the Cyfronet bucket already exists) and add it to
MODEL_MANIFEST.

5.5 Smaller items

6. Engineering-practice gaps

6.1 A red main still deploys 🚨

deploy.yml triggers on push: [main] and its build job runs only pnpm build — no needs on
ci.yml, no gate steps of its own. The two workflows race independently, so a push that breaks
lint, typecheck, validate:eval, or all 808 tests still publishes to GitHub Pages. Fix: either
trigger deploy on workflow_run (workflows: [CI], types: [completed], conditioned on success), or
run the gate inside deploy.yml. Worth confirming branch protection requires the CI checks too.

6.2 Dependabot covers one of three ecosystems

.github/dependabot.yml has a single npm entry at /. Missing:
package-ecosystem: "github-actions" (the @v4/@v5 action pins never update) and
package-ecosystem: "gradle" for the bench/android/* apps.

6.3 The drift guard nobody runs

bench/android/full-app/app/src/test/java/.../KotlinMatcherAgreementTest.kt is the only thing
preventing the TS and Kotlin matchers from silently diverging — and no workflow invokes Gradle. Add a
job running ./gradlew :app:test for bench/android/full-app (path-filtered to bench/android/**
so it doesn't slow the main gate). #160's checklist already assumes this test will catch
multi-quantity divergence; today it can't, because it never executes.

6.4 No coverage measurement

pnpm test runs without --coverage and there's no threshold. Not urgent (coverage is visibly
decent) but a reported non-blocking number would show new modules landing untested.

6.5 Green gates, hidden bugs — a gap worth naming

All 5 gates pass, and none of B1–B10 is detectable by any of them. The gates verify form (format,
types, schema, unit behavior). Nothing verifies physics. One cheap addition would change that: a
physics golden-file test — ~30 canonical (query → expected number ± tolerance) pairs, spanning
both inverse quantities and every accepted target unit, asserted against the real WASM the way
compute.smoke.test.ts already does. B1, B2, B4, B7 all fail such a test immediately. This is the
single best return on effort in this section.

7. docs/ — diagnosis before the plan

Measured today:

  • 32 markdown files, 9,586 lines, 804 KB in docs/.
  • 9 of the 31 content docs are absent from docs/README.md's index — and they are the newest and
    by far the largest: android-asr-runtime-bench.md (955 lines), unit-pronunciation-asr.md (627),
    android-full-app-spike.md (617), nemo-parakeet-comparison.md (576), whisper-model-bench.md
    (492), android-datagen-bench.md (408), tts-eval-1000-pl.md (219), tts-chatterbox-pl-clone.md
    (141), forced-alignment-setup.md (125). The index was last touched 2026-07-18; all nine landed
    07-19…07-30. ~43% of docs/ by volume is unindexed, and it's the half a reader most needs.
  • Named by run, not by question: tts-eval-1000 / -v2 / -v3 / -pl / tts-eval-audio /
    tts-chatterbox-pl-clone are six documents answering roughly one question. Same for the ASR side:
    asr-model-comparison, apple-silicon-benchmark, whisper-model-bench,
    nemo-parakeet-comparison, android-asr-runtime-bench, android-datagen-bench. A reader asking
    "which ASR model should I use, and how fast is it?" must read six documents in date order and
    reconcile them — which docs/README.md states outright as the intended reading protocol.
  • eval/results/ is 124 MB across 590 tracked files (357 JSON + 233 .log; the git pack is only
    10.2 MB, so clone cost is fine — the cost is navigation and grep noise). 114 MB / 484 files of
    that sit in two directories named after Slurm job IDs    : whisper-bench-2805165,
    whisper-bench-2807345. Nothing but one doc explains what those numbers mean.
  • nlp-libraries-polish.md has zero inbound references from code or any other doc.
    user-guide.md is a 10-line "coming soon" stub, linked from the README as a top-level entry.

The actual root cause

/eval/audio/ is gitignored (.gitignore:29). Zero .wav/.mp3/.flac files are tracked
anywhere in the repo. #160 §7 reports 289 recordings across 6 speaker sets — all of them living
only on individual laptops.

So every benchmark number in all 9,586 lines of docs/ was computed from inputs no second person
can obtain. Nothing is re-runnable, so nothing can be updated — each new run has to become a
new document, and the old one has to stay, because it's the only surviving record of its own
inputs. The docs sprawl isn't a documentation-hygiene failure. It's the observable symptom of an
unversioned, unshared corpus, and it will regenerate itself no matter how carefully docs/ is
tidied.

Which means the reorganization has to start with the corpus. Your instinct — new corpus, rerun,
delete most of docs/ — is right, and this is the ordering that makes it stick.

8. Proposed reorganization

Phase 0 — freeze, don't hoard

Tag the current state (git tag docs-archive-2026-07-30 && git push --tags) and say so in the new
index. Then delete freely: git history keeps everything, and the tag makes it findable. Do not
create docs/archive/ — that's the same 9,586 lines with a prefix, and it will still be in every
grep. One rule for the whole effort: harvest conclusions first, then delete the document.

Phase 1 — build corpus v1 (the load-bearing step)

corpus/v1/
  manifest.jsonl      # one row per utterance (tracked)
  SHA256SUMS          # audio checksums (tracked)
  README.md           # how to fetch, how to extend, speaker consent notes
  audio/              # gitignored; fetched by scripts/corpus-fetch.ts

Each manifest row: id, lang, speaker, canonical text, display (as-read), slotTruth,
intentId → eval/intents.jsonl, phenomena[] tags, sha256, durationMs, device. This is
eval/RECORDING.datagen.md's existing convention, promoted to the primary artifact.

Host the audio on infrastructure you already have. The Cyfronet S3 bucket,
scripts/mirror-upload-s3.sh, and docs/model-hosting-cyfronet.md already exist for model weights.
Publish aidedx-corpus-v1.tar.zst + SHA256SUMS there; scripts/corpus-fetch.ts downloads and
verifies. Cost: ~zero new infrastructure. Benefit: the reproducibility gap closes permanently.

What to record — drive the sentence list from the phenomena that have actually broken things,
not from a fresh brainstorm:

Sizing: ~150 utterances × 4 speakers × 2 languages is a realistic ceiling for a recording campaign
and comfortably more discriminating than 122 text rows. Put the target in corpus/v1/README.md so
v2 can be compared against it.

Pair it with the text-only benchmark. #160 §8's pnpm bench:nlu is the other half: corpus v1
measures the audio path, bench:nlu measures the language path in CI in seconds, with a mutation
layer that generates B4's whole family automatically. Land bench:nlu first — it's cheaper, it
gates every PR, and B4 proves the need.

Phase 2 — one runner, one result schema

Today: asr-batch.mjs, asr-transcribe.mjs, asr-transcribe-manifest.mjs,
sherpa-onnx-transcribe.mjs, asr-score-slots.mjs, asr-score-slots-generic.mjs,
e2e-audio-intents.ts, e2e-audio-intents-datagen.ts — four pairs of near-duplicates, each pair
born when a new corpus shape appeared. Collapse to:

  • scripts/bench-asr.ts --corpus v1 --runner <whisper-web|whispercpp|parakeet|sherpa|desktop>
    → eval/results/v1/<runner>/<date>.json, one documented schema, with the corpus version and
    SHA256SUMS digest recorded inside every result file;
  • scripts/bench-report.ts → regenerates the comparison tables in docs/benchmarks.md.

Then results are data, docs are interpretation, and -v2/-v3/-generic suffixes stop breeding.

Phase 3 — rerun, harvest, delete

Rerun the runners that still matter (whisper-small q8 web, whisper.cpp Android, Parakeet, desktop
whisper) on corpus v1. Then, per document, harvest the conclusion into its new home and delete:

Delete after harvesting Conclusion goes to
asr-model-comparison.md, apple-silicon-benchmark.md, whisper-model-bench.md, nemo-parakeet-comparison.md, android-asr-runtime-bench.md, android-datagen-bench.md docs/benchmarks.md (current numbers) + one ADR ("why whisper-small on the CPU tier")
tts-eval-1000.md, -v2, -v3, -pl, tts-eval-audio.md, tts-chatterbox-pl-clone.md one ADR: "TTS-synthesized audio: what it can and can't measure" — the single durable finding across all six
unit-pronunciation-asr.md (627 lines, 11 inbound code refs) the phenomena list in corpus/v1/README.md — that's what those 11 references actually want
nlp-libraries-polish.md (zero inbound refs) one ADR: "why no NLP library" (2 paragraphs)
design.md delete; its own index row says "superseded"
user-guide.md (10-line stub) delete, or write it
eval/results/whisper-bench-2805165/, -2807345/ (114 MB, 484 files) delete; unreproducible by construction, and the tag preserves them
Keep as living reference (1:1 with shipped code)
nlu.md, aliases.md, wasm.md, answer-pipeline.md, status-panel-design.md, model-hosting-cyfronet.md, local-model-cache.md, development.md, athena-setup.md, phonetic-corrector.md

Plus three new: docs/benchmarks.md (living; generated tables + interpretation),
docs/corpus.md (fetch/extend/consent), docs/decisions/NNNN-*.md (ADRs — deterministic-
first NLU, whisper-small, COOP/COEP service worker §5.3, Cyfronet mirror, no-NLP-library, TTS's
limits; #160 §9 will add the matcher-formalization decision).

Target: ~14 docs, ~3,000 lines, down from 31 / 9,586 — with every benchmark number reproducible.

Phase 4 — make staleness structurally impossible

This is the part that stops it recurring, and it's small:

  1. scripts/check-docs-index.ts in the static-analysis job (~40 lines): every docs/*.md
    appears in docs/README.md; every index link resolves; every doc has a status header. This alone
    makes the 9-missing-docs failure impossible.
  2. Status header on every doc: 
    Status: living | decision | superseded-by <file>
Corpus: v1        Last verified: 2026-08-xx
    CI asserts every benchmarks-class doc names a corpus version.
  3. House rule: no benchmark number in prose without a corpus tag. A number without a corpus
    version is unfalsifiable, and unfalsifiable numbers are what made these 31 files un-deletable.
  4. CLAUDE.md amendment. Its current rule ("research/spike issues must land findings in a
    committed doc before closing") is what produced the sprawl — a good rule with no update path. Add:
    findings update an existing living doc or an ADR; a new docs/*.md per run is not the default.

Phase 5 — a note on the audience

For a physicist reader, the docs that matter are (a) what the numbers mean and how far to trust them,
(b) which materials/particles/programs libdedx actually covers, and (c) what the app assumed. (b) and
(c) are exactly what B3 and B7 get wrong in the product, and neither has a doc. Worth adding
docs/coverage.md — generated from the alias tables and libdedx's own availability lists — listing
what can and cannot be asked about. That's ~50 lines of codegen, it's the answer to B3's "we don't
have stainless steel", and it's the page a new user will actually read.

9. Cross-references

10. Suggested order

Updated 2026-07-31 after the re-audit (see the status block at the top). C* items are the
re-audit's new findings; B*/§* are the original ones.

Done — the original P1s (#166, #167, #168):

Now — silent wrong numbers found in the re-audit:

Then — misleading, not silently wrong:

Then — instrument (each of these would have caught something above):

Then — libdedx forward compatibility (do before either new quantity lands):

Then — process (each is < 1 hour):

Then — corpus and docs (§8):

  • Phase 0 tag → Phase 1 corpus v1 + corpus-fetch.ts → Phase 2 single runner → Phase 3 rerun,
    harvest, delete → Phase 4 check-docs-index.ts + status headers + CLAUDE.md amendment
  • docs/coverage.md generated from the alias tables (§8 Phase 5) ([issue-163] §8 Phase 5: add generated docs/coverage.md #216)

Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requesteval

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions