Skip to content

MinGram across the grid, and the bnd_wpd_extcaps sweep - #11

Merged
sanderland merged 3 commits into
sanderland:claude/fineweb-space-neighbors-k10ufwfrom
cimeister:claude/mingram-grid-extcaps
Aug 4, 2026
Merged

MinGram across the grid, and the bnd_wpd_extcaps sweep#11
sanderland merged 3 commits into
sanderland:claude/fineweb-space-neighbors-k10ufwfrom
cimeister:claude/mingram-grid-extcaps

Conversation

@cimeister

Copy link
Copy Markdown

Three commits, cut from the current tip so this fast-forwards.

MinGram now covers every language. The grid had MinGram for English plain and bnd_wpd only, so make_intrinsic_table.py dropped the trainer entirely, since a trainer earns a column group only when it covers the same languages as the leading one. 18 new cells complete plain, bnd_w, bnd_wp and bnd_wpd for en, de, fi, ru and ar. table_intrinsic.tex goes from 46 to 56 of 72 cells, and the two trainers can now be read against each other. The scheme ordering holds under both: MinGram eval means of $-9.85$, $-2.70$, $-0.96$ against BPE's $-9.30$, $-2.18$, $-0.46$.

Every corpus was already cached, so only the trainer ran: 1.48 node-hours for the 16 non-English cells.

bnd_wpd_extcaps measures as no different from bnd_wpd_caps downstream. Three seeds, every setting matched to the committed BPE runs, paired by seed:

comparison mean SE t(2)
vs plain +0.006026 0.000128 47.05
vs bnd_wpd +0.000709 0.000268 2.64
vs bnd_wpd_caps +0.000247 0.000233 1.06

The difference against bnd_wpd_caps is smaller than either arm's seed standard deviation, and compression agrees (+0.75% against +0.72% on Goldfish). The commit adding the arm said the probe corpus could not answer whether the shared word entry is reused; at this scale it makes no measurable difference either way.

Two things you should look at rather than take on trust.

fineweb_ru_5gb_plain_mingram_v34685 is vocabulary 34,684, one short, and is committed as it is. The cause is in the MinGram trainer: the target check at script_bpe/tokenizers/mingram/trainer.py:57-64 sits outside the inner EM loop, so once the size reaches the target the remaining sub-steps still prune. Its log shows EM 2.1. Model size 34,685 then EM 2.2. Model size 34,684. Any MinGram cell can overshoot by up to num_em_iterations - 1; the rest landed exactly by luck. merge_manifests.py correctly refuses to rewrite manifest.json for this grid, and I left that refusal in place rather than passing --no-check, so manifest.json is untouched here.

The Goldfish measurement was run twice and the first was thrown away. Run as python marker_experiments/downstream/eval_goldfish.py, sys.path[0] is the script's directory, which holds no marker_experiments package, so the import resolved through the editable install to a different checkout on a different branch. It measured 10 cells against that branch's pretokenizer before raising KeyError: 'ExtCapsBoundaryScriptPretokenizer' on the eleventh. Those 10 were already written and would have looked ordinary in review. They were discarded and the measurement rerun with PYTHONPATH pinned. Worth knowing if you ever run that script from a worktree.

submit_extcaps.sh also picks up four fixes already made in the MinGram launcher: pinned SMOKE and TAG_SUFFIX, a clean-tree guard that uses git status --porcelain rather than git diff so it sees untracked files, a squeue failure that aborts instead of yielding an empty in-flight list, and a required ACCOUNT.

The grid had MinGram for English plain and bnd_wpd only, so the intrinsic table dropped
the trainer entirely: a trainer earns a column group only when it covers the same
languages as the leading one. These 18 cells complete plain, bnd_w, bnd_wp and bnd_wpd for
en, de, fi, ru and ar.

Jobs 2994413 and 2997267/69/71/73 on infra01, one job per language. Every pretokenized
corpus was already cached, so only the trainer ran: 15m16s to 33m58s per job, 1.48
node-hours for all 16 of the non-English cells.

17 of the 18 land at vocabulary 34,685 with 0 round-trip failures.

fineweb_ru_5gb_plain_mingram_v34685 lands at 34,684, one token short, and is committed as
it is rather than dropped or retrained. The cause is in the MinGram trainer: the target
check at script_bpe/tokenizers/mingram/trainer.py:57-64 sits outside the inner EM loop, so
once the size reaches the target the remaining sub-steps still prune. Its log shows
`EM 2.1. Model size 34,685` followed by `EM 2.2. Model size 34,684`. Any MinGram cell can
overshoot by up to num_em_iterations - 1 tokens; the others landed exactly by luck.

merge_manifests.py therefore refuses to rewrite manifest.json for this grid, reporting
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685,
'plain': 34684}. That refusal is correct and is left in place; --no-check is not used,
since bypassing it would write a manifest asserting a match it does not have. The per-cell
fragments carry the record and the tables read eval_goldfish.json.
Moving the caps code outside the boundary markers does not measurably change the
downstream loss against bnd_wpd_caps, which puts it inside.

Validation bits per true byte, three seeds, every setting matched to the committed BPE
runs and paired by seed:

    plain             0.885315 (sd 0.000312)
    bnd_wpd           0.879999 (sd 0.000487)
    bnd_wpd_caps      0.879536 (sd 0.000736)
    bnd_wpd_extcaps   0.879290 (sd 0.000458)

Paired, positive meaning extcaps is lower:

    vs plain          +0.006026  SE 0.000128  t(2) = 47.05
    vs bnd_wpd        +0.000709  SE 0.000268  t(2) =  2.64
    vs bnd_wpd_caps   +0.000247  SE 0.000233  t(2) =  1.06

The difference against bnd_wpd_caps is smaller than either arm's seed standard deviation.
Compression agrees: 3.7464 against 3.7456 chars/token on the held-out English slice, and
+0.75% against +0.72% on Goldfish. The commit that added the arm noted the probe corpus
could not answer whether the shared word entry is reused; at this scale the answer is that
it makes no measurable difference either way.

Tokenizer trained by job 2992690, vocabulary 34,685, 0 round-trip failures. CORE was not
scored: core_prefix_check.py reports 399/512 aborts, the same as bnd_wpd and
bnd_wpd_caps, so the sweep ran with CORE_SAFE_ARMS="" on bits-per-byte alone.

submit_extcaps.sh carries four fixes, each already made in the MinGram launcher:

- SMOKE and TAG_SUFFIX are pinned in the job. --wrap runs under the submitting shell's
  environment, so a leftover SMOKE=1 would have made every seed train 20 iterations, tag
  itself _smoke, and still print a result block that collect_results.py parses into a row.
- The clean-tree guard uses git status --porcelain; git diff does not report untracked
  files, so it passed while a new file the job runs was described by no commit.
- A squeue failure aborts rather than defaulting to an empty in-flight list, which would
  have submitted duplicates sharing a checkpoint directory and a log.
- ACCOUNT is required with no default, replacing a hardcoded one.
eval_goldfish.py measures only keys missing from eval_goldfish.json, so this adds the 18
MinGram cells and bnd_wpd_extcaps and leaves the existing 133 untouched. 152 keys, 0
round-trip failures across all 19.

table_intrinsic.tex goes from 46 to 56 of 72 cells. The MinGram half previously had Korean
and two English cells; it now has plain, bnd_w, bnd_wp and bnd_wpd for all six languages,
so the two trainers can be read against each other rather than one being a fragment. The
scheme ordering holds under both trainers: bnd_w is the largest loss, bnd_wp smaller,
bnd_wpd close to plain, with MinGram means of -9.85, -2.70 and -0.96 on eval against BPE's
-9.30, -2.18 and -0.46.

Both runs of this measurement are in the history of this branch for a reason. The first
was discarded: run as `python marker_experiments/downstream/eval_goldfish.py`, sys.path[0]
is the script's directory, which holds no marker_experiments package, so the import
resolved through the editable install to /users/cmeister747/script_tok, a different branch
whose boundary_pretokenizer.py has no ExtCapsBoundaryScriptPretokenizer. It measured 10
cells against that branch's pretokenizer before raising KeyError on the eleventh. Those 10
were written to eval_goldfish.json and would have looked ordinary in review. They were
discarded by restoring the file, and the measurement rerun with
PYTHONPATH="$REPO:$REPO/eval/py-nanochat".
@cimeister
cimeister force-pushed the claude/mingram-grid-extcaps branch from e169aab to 4c60a60 Compare August 4, 2026 08:13
@sanderland
sanderland merged commit fb66f58 into sanderland:claude/fineweb-space-neighbors-k10ufw Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants