MinGram across the grid, and the bnd_wpd_extcaps sweep - #11
Merged
sanderland merged 3 commits intoAug 4, 2026
Conversation
The grid had MinGram for English plain and bnd_wpd only, so the intrinsic table dropped
the trainer entirely: a trainer earns a column group only when it covers the same
languages as the leading one. These 18 cells complete plain, bnd_w, bnd_wp and bnd_wpd for
en, de, fi, ru and ar.
Jobs 2994413 and 2997267/69/71/73 on infra01, one job per language. Every pretokenized
corpus was already cached, so only the trainer ran: 15m16s to 33m58s per job, 1.48
node-hours for all 16 of the non-English cells.
17 of the 18 land at vocabulary 34,685 with 0 round-trip failures.
fineweb_ru_5gb_plain_mingram_v34685 lands at 34,684, one token short, and is committed as
it is rather than dropped or retrained. The cause is in the MinGram trainer: the target
check at script_bpe/tokenizers/mingram/trainer.py:57-64 sits outside the inner EM loop, so
once the size reaches the target the remaining sub-steps still prune. Its log shows
`EM 2.1. Model size 34,685` followed by `EM 2.2. Model size 34,684`. Any MinGram cell can
overshoot by up to num_em_iterations - 1 tokens; the others landed exactly by luck.
merge_manifests.py therefore refuses to rewrite manifest.json for this grid, reporting
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685,
'plain': 34684}. That refusal is correct and is left in place; --no-check is not used,
since bypassing it would write a manifest asserting a match it does not have. The per-cell
fragments carry the record and the tables read eval_goldfish.json.
Moving the caps code outside the boundary markers does not measurably change the
downstream loss against bnd_wpd_caps, which puts it inside.
Validation bits per true byte, three seeds, every setting matched to the committed BPE
runs and paired by seed:
plain 0.885315 (sd 0.000312)
bnd_wpd 0.879999 (sd 0.000487)
bnd_wpd_caps 0.879536 (sd 0.000736)
bnd_wpd_extcaps 0.879290 (sd 0.000458)
Paired, positive meaning extcaps is lower:
vs plain +0.006026 SE 0.000128 t(2) = 47.05
vs bnd_wpd +0.000709 SE 0.000268 t(2) = 2.64
vs bnd_wpd_caps +0.000247 SE 0.000233 t(2) = 1.06
The difference against bnd_wpd_caps is smaller than either arm's seed standard deviation.
Compression agrees: 3.7464 against 3.7456 chars/token on the held-out English slice, and
+0.75% against +0.72% on Goldfish. The commit that added the arm noted the probe corpus
could not answer whether the shared word entry is reused; at this scale the answer is that
it makes no measurable difference either way.
Tokenizer trained by job 2992690, vocabulary 34,685, 0 round-trip failures. CORE was not
scored: core_prefix_check.py reports 399/512 aborts, the same as bnd_wpd and
bnd_wpd_caps, so the sweep ran with CORE_SAFE_ARMS="" on bits-per-byte alone.
submit_extcaps.sh carries four fixes, each already made in the MinGram launcher:
- SMOKE and TAG_SUFFIX are pinned in the job. --wrap runs under the submitting shell's
environment, so a leftover SMOKE=1 would have made every seed train 20 iterations, tag
itself _smoke, and still print a result block that collect_results.py parses into a row.
- The clean-tree guard uses git status --porcelain; git diff does not report untracked
files, so it passed while a new file the job runs was described by no commit.
- A squeue failure aborts rather than defaulting to an empty in-flight list, which would
have submitted duplicates sharing a checkpoint directory and a log.
- ACCOUNT is required with no default, replacing a hardcoded one.
eval_goldfish.py measures only keys missing from eval_goldfish.json, so this adds the 18 MinGram cells and bnd_wpd_extcaps and leaves the existing 133 untouched. 152 keys, 0 round-trip failures across all 19. table_intrinsic.tex goes from 46 to 56 of 72 cells. The MinGram half previously had Korean and two English cells; it now has plain, bnd_w, bnd_wp and bnd_wpd for all six languages, so the two trainers can be read against each other rather than one being a fragment. The scheme ordering holds under both trainers: bnd_w is the largest loss, bnd_wp smaller, bnd_wpd close to plain, with MinGram means of -9.85, -2.70 and -0.96 on eval against BPE's -9.30, -2.18 and -0.46. Both runs of this measurement are in the history of this branch for a reason. The first was discarded: run as `python marker_experiments/downstream/eval_goldfish.py`, sys.path[0] is the script's directory, which holds no marker_experiments package, so the import resolved through the editable install to /users/cmeister747/script_tok, a different branch whose boundary_pretokenizer.py has no ExtCapsBoundaryScriptPretokenizer. It measured 10 cells against that branch's pretokenizer before raising KeyError on the eleventh. Those 10 were written to eval_goldfish.json and would have looked ordinary in review. They were discarded by restoring the file, and the measurement rerun with PYTHONPATH="$REPO:$REPO/eval/py-nanochat".
cimeister
force-pushed
the
claude/mingram-grid-extcaps
branch
from
August 4, 2026 08:13
e169aab to
4c60a60
Compare
sanderland
merged commit Aug 4, 2026
fb66f58
into
sanderland:claude/fineweb-space-neighbors-k10ufw
1 check passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, cut from the current tip so this fast-forwards.
MinGram now covers every language. The grid had MinGram for English$-9.85$ , $-2.70$ , $-0.96$ against BPE's $-9.30$ , $-2.18$ , $-0.46$ .
plainandbnd_wpdonly, somake_intrinsic_table.pydropped the trainer entirely, since a trainer earns a column group only when it covers the same languages as the leading one. 18 new cells completeplain,bnd_w,bnd_wpandbnd_wpdfor en, de, fi, ru and ar.table_intrinsic.texgoes from 46 to 56 of 72 cells, and the two trainers can now be read against each other. The scheme ordering holds under both: MinGram eval means ofEvery corpus was already cached, so only the trainer ran: 1.48 node-hours for the 16 non-English cells.
bnd_wpd_extcapsmeasures as no different frombnd_wpd_capsdownstream. Three seeds, every setting matched to the committed BPE runs, paired by seed:plainbnd_wpdbnd_wpd_capsThe difference against
bnd_wpd_capsis smaller than either arm's seed standard deviation, and compression agrees (+0.75% against +0.72% on Goldfish). The commit adding the arm said the probe corpus could not answer whether the shared word entry is reused; at this scale it makes no measurable difference either way.Two things you should look at rather than take on trust.
fineweb_ru_5gb_plain_mingram_v34685is vocabulary 34,684, one short, and is committed as it is. The cause is in the MinGram trainer: the target check atscript_bpe/tokenizers/mingram/trainer.py:57-64sits outside the inner EM loop, so once the size reaches the target the remaining sub-steps still prune. Its log showsEM 2.1. Model size 34,685thenEM 2.2. Model size 34,684. Any MinGram cell can overshoot by up tonum_em_iterations - 1; the rest landed exactly by luck.merge_manifests.pycorrectly refuses to rewritemanifest.jsonfor this grid, and I left that refusal in place rather than passing--no-check, somanifest.jsonis untouched here.The Goldfish measurement was run twice and the first was thrown away. Run as
python marker_experiments/downstream/eval_goldfish.py,sys.path[0]is the script's directory, which holds nomarker_experimentspackage, so the import resolved through the editable install to a different checkout on a different branch. It measured 10 cells against that branch's pretokenizer before raisingKeyError: 'ExtCapsBoundaryScriptPretokenizer'on the eleventh. Those 10 were already written and would have looked ordinary in review. They were discarded and the measurement rerun withPYTHONPATHpinned. Worth knowing if you ever run that script from a worktree.submit_extcaps.shalso picks up four fixes already made in the MinGram launcher: pinnedSMOKEandTAG_SUFFIX, a clean-tree guard that usesgit status --porcelainrather thangit diffso it sees untracked files, asqueuefailure that aborts instead of yielding an empty in-flight list, and a requiredACCOUNT.