MinGram downstream: bnd_w, and the placement ablation - #13
Merged
sanderland merged 1 commit intoAug 4, 2026
Conversation
Nine runs, three arms by three seeds, jobs 2998893-95, every setting matched to the
committed BPE and MinGram runs. results_mingram.tsv goes from 6 rows to 15, and the main
table's MinGram column for bnd_w is filled in rather than blank.
Validation bits per true byte, seeds 0 to 2:
plain 0.883694 (sd 0.000808)
bnd_wpd 0.880507 (sd 0.000625)
bnd_wpd_caps 0.880076 (sd 0.000105)
bnd_w 0.876321 (sd 0.000251)
bnd_wpd_extcaps 0.876042 (sd 0.000232)
Paired by seed, positive meaning the second arm is lower, MinGram against BPE:
plain vs bnd_wpd +0.003187 t(2)=15.2 | +0.005317 t(2)=16.7
plain vs bnd_wpd_caps +0.003618 t(2)= 7.8 | +0.005779 t(2)=23.6
plain vs bnd_wpd_extcaps +0.007652 t(2)=12.8 | +0.006026 t(2)=47.0
caps vs extcaps +0.004034 t(2)=26.5 | +0.000247 t(2)= 1.1
The placement ablation is out of the downstream tables by your choice in 90cbf37, so those
two arms appear only in results_mingram.tsv. The number is worth a look before it stays
dropped: the caps-code placement is indistinguishable under BPE, t(2)=1.1, and under
MinGram placing the code outside the markers is better by 0.004034 at t(2)=26.5, larger
than the whole plain-to-bnd_wpd effect. With the code outside, the lower-cased span is a
suffix of the cased form, so one word entry serves both, and MinGram prunes an overshot
vocabulary with EM rather than growing greedily.
bnd_w lands at 0.876321, close to extcaps, but its byte factor is 0.8630 against plain's
1.0006, so it emits more tokens per byte and buys part of that with extra forward passes.
caps against extcaps is not confounded that way: 0.8426 against 0.8444.
Three seeds per arm. The t values are large because the seed spread is small.
Checked before reporting, since three earlier sweeps produced plausible numbers that were
wrong: the gain is in the raw bpb (1.0375 against 1.0444), not the byte-factor correction,
whose values differ by 0.21%; each arm has its own token_bytes.pt; every run evaluated
--step 2553, the checkpoint it had just trained; each tokenizer's sha256 matches what the
in-job gate recorded; and the per-seed data permutations differ while being shared across
arms, which is what makes the pairing valid.
smoke_test.py takes --corpus so that --require-matched-vocab compares within one corpus and
trainer. Without it the pattern matched all 75 MinGram tokenizers and demanded a single
vocabulary across every corpus, so fineweb_ru_5gb_plain_mingram at 34,684 aborted all nine
runs before any training. Verified both ways: the English arms pass, and asking for
fineweb_ru_5gb still refuses.
manifest.json is merged with --no-check so the two new tokenizers appear; the check still
reports that Russian cell.
sanderland
merged commit Aug 4, 2026
43886d6
into
sanderland:claude/fineweb-space-neighbors-k10ufw
1 check passed
sanderland
added a commit
that referenced
this pull request
Aug 5, 2026
A reproduction directory in the shape of the other three papers, and one
pretokenizer configured by options instead of three classes.
./paper_utils/boundary/run_all_experiments.sh
regenerates every table from the caches committed under paper/generated/ -- no
GPU, no trained tokenizer, about a second. GRID=1 retrains the tokenizer grid,
DOWNSTREAM=1 reruns the LM sweep and exits with a usable message where there is
no CUDA.
The pretokenizer
BoundaryScriptPretokenizer is the only class. Case codes always sit outside
the span's markers and are applied only to a span that carries case; the
inside-the-markers placement and the version with no case test are gone.
Options replace the subclasses:
boundary_targets ("punct",) by default -- "word" is not a member, since
word spans are always delimited
shift_code <^> title case
caps_code <^^> all caps
min_caps_length shortest span <^^> may cover, 2
single_char_shift whether "A" is shift-coded, True
The defaults reproduce the reported arm exactly, so the committed caches keep
describing the default configuration. Verified before converting anything:
encode and decode agree with the old classes on 263 texts x 6 variants x 3
digit_handling settings, 0 mismatches. All 164 trained tokenizers embed their
config, so they were rewritten in place -- 84 converted, 56 deleted as cells
of the two removed schemes, 24 baselines untouched -- and each converted file
was checked by loading it and hash-matching the freshly built variant.
tests/test_boundary.py merges the two old suites and covers each option, 1412
tests.
The tables
table_intrinsic_main.tex and table_downstream_main.tex regenerate
byte-identical apart from the header naming the new script path.
bnd_wpd_extcapsfix is renamed bnd_wpd_caps across every cache. The
full-corpus cells are dropped: the paper reports the quick sample, the full
grid was 48 of a possible 108, and one cell (fineweb_ru_5gb_plain_mingram)
had landed at 34,684 against 34,685 everywhere else, which made
merge_manifests refuse to write. That cell is one token short because the
MinGram trainer tests the target outside its inner EM loop
(script_bpe/tokenizers/mingram/trainer.py:57-64), which is not fixed here.
make_tex_tables reads round-trip failures from eval_goldfish.json, which
measures every cell, rather than from manifest entries only a cell trained
with an --eval-texts slice ever had.
make_example_table.py generates the worked pre-tokenization example. It runs
the pretokenizers directly, before any vocabulary is learned, so it needs no
trained tokenizer and no corpus, and it reproduces the paper's table byte for
byte with the counts computed rather than typed.
Picked up from #13, which merged after this branch was cut
results_mingram.tsv gains nine runs, so the MinGram column has no missing
cell: bnd_w at 0.8763 and the case codes at 0.8760. Its bnd_wpd_extcaps is
this branch's bnd_wpd_caps; its bnd_wpd_caps was the inside-the-markers
placement and is dropped as the BPE one was. Also smoke_test.py --corpus,
without which --require-matched-vocab compares across every corpus at once
and one short cell anywhere aborts every run.
Deleted
The superseded FineWiki grids, probes and prototypes; 62 MB of FineWiki 32k
tokenizers; the root session logs; and the bits-per-byte-against-compression
figure, which is not in the paper. The 170 MB of matched tokenizers, the
generated .tex and the per-cell manifest fragments are untracked but kept
locally -- manifest.json holds exactly the fragments' union, so tracking both
committed the same 60 entries twice. What stays in git is 480 KB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7F6Ac72HbPG15ExjkbFFu
sanderland
added a commit
that referenced
this pull request
Aug 5, 2026
A reproduction directory in the shape of the other three papers, and one
pretokenizer configured by options instead of three classes.
./paper_utils/boundary/run_all_experiments.sh
regenerates every table from the caches committed under paper/generated/ -- no
GPU, no trained tokenizer, about a second. GRID=1 retrains the tokenizer grid,
DOWNSTREAM=1 reruns the LM sweep and exits with a usable message where there is
no CUDA.
The pretokenizer
BoundaryScriptPretokenizer is the only class. Case codes always sit outside
the span's markers and are applied only to a span that carries case; the
inside-the-markers placement and the version with no case test are gone.
Options replace the subclasses:
boundary_targets ("punct",) by default -- "word" is not a member, since
word spans are always delimited
shift_code <^> title case
caps_code <^^> all caps
min_caps_length shortest span <^^> may cover, 2
single_char_shift whether "A" is shift-coded, True
The defaults reproduce the reported arm exactly, so the committed caches keep
describing the default configuration. Verified before converting anything:
encode and decode agree with the old classes on 263 texts x 6 variants x 3
digit_handling settings, 0 mismatches. All 164 trained tokenizers embed their
config, so they were rewritten in place -- 84 converted, 56 deleted as cells
of the two removed schemes, 24 baselines untouched -- and each converted file
was checked by loading it and hash-matching the freshly built variant.
tests/test_boundary.py merges the two old suites and covers each option, 1412
tests.
The tables
table_intrinsic_main.tex and table_downstream_main.tex regenerate
byte-identical apart from the header naming the new script path.
bnd_wpd_extcapsfix is renamed bnd_wpd_caps across every cache. The
full-corpus cells are dropped: the paper reports the quick sample, the full
grid was 48 of a possible 108, and one cell (fineweb_ru_5gb_plain_mingram)
had landed at 34,684 against 34,685 everywhere else, which made
merge_manifests refuse to write. That cell is one token short because the
MinGram trainer tests the target outside its inner EM loop
(script_bpe/tokenizers/mingram/trainer.py:57-64), which is not fixed here.
make_tex_tables reads round-trip failures from eval_goldfish.json, which
measures every cell, rather than from manifest entries only a cell trained
with an --eval-texts slice ever had.
make_example_table.py generates the worked pre-tokenization example. It runs
the pretokenizers directly, before any vocabulary is learned, so it needs no
trained tokenizer and no corpus, and it reproduces the paper's table byte for
byte with the counts computed rather than typed.
Downstream
run_arms.sh is a plain sequential driver: it trains the matched tokenizers,
gates them, precomputes the byte factors, and then calls
run_downstream_eval.py once per arm x seed. Scheduling is left to whoever
reproduces it -- each model is one invocation of that script, and runs are
skip-if-done, so cells can be submitted anywhere and a re-run fills in the
rest. The CSCS Clariden job scripts, their site-specific environment and the
README describing that flow are deleted, as are the preflight and
corpus-prebuilding helpers that only existed to pack those jobs.
results_mingram.tsv takes the nine runs from #13, so the MinGram column has
no missing cell: bnd_w at 0.8763 and the case codes at 0.8760.
Deleted
The superseded FineWiki grids, probes and prototypes; 62 MB of FineWiki 32k
tokenizers; the root session logs; and the bits-per-byte-against-compression
figure, which is not in the paper. The 170 MB of matched tokenizers, the
generated .tex and the per-cell manifest fragments are untracked but kept
locally -- manifest.json holds exactly the fragments' union, so tracking both
committed the same 60 entries twice. What stays in git is 460 KB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7F6Ac72HbPG15ExjkbFFu
sanderland
added a commit
that referenced
this pull request
Aug 5, 2026
A reproduction directory in the shape of the other three papers, and one
pretokenizer configured by options instead of three classes.
./paper_utils/boundary/run_all_experiments.sh
regenerates every table from the caches committed under paper/generated/ -- no
GPU, no trained tokenizer, about a second. GRID=1 retrains the tokenizer grid,
DOWNSTREAM=1 reruns the LM sweep and exits with a usable message where there is
no CUDA.
The pretokenizer
BoundaryScriptPretokenizer is the only class. Case codes always sit outside
the span's markers and are applied only to a span that carries case; the
inside-the-markers placement and the version with no case test are gone.
Options replace the subclasses:
boundary_targets ("punct",) by default -- "word" is not a member, since
word spans are always delimited
shift_code <^> title case
caps_code <^^> all caps
min_caps_length shortest span <^^> may cover, 2
single_char_shift whether "A" is shift-coded, True
The defaults reproduce the reported arm exactly, so the committed caches keep
describing the default configuration. Verified before converting anything:
encode and decode agree with the old classes on 263 texts x 6 variants x 3
digit_handling settings, 0 mismatches. All 164 trained tokenizers embed their
config, so they were rewritten in place -- 84 converted, 56 deleted as cells
of the two removed schemes, 24 baselines untouched -- and each converted file
was checked by loading it and hash-matching the freshly built variant.
tests/test_boundary.py merges the two old suites and covers each option, 1412
tests.
The tables
table_intrinsic_main.tex and table_downstream_main.tex regenerate
byte-identical apart from the header naming the new script path.
bnd_wpd_extcapsfix is renamed bnd_wpd_caps across every cache. The
full-corpus cells are dropped: the paper reports the quick sample, the full
grid was 48 of a possible 108, and one cell (fineweb_ru_5gb_plain_mingram)
had landed at 34,684 against 34,685 everywhere else, which made
merge_manifests refuse to write. That cell is one token short because the
MinGram trainer tests the target outside its inner EM loop
(script_bpe/tokenizers/mingram/trainer.py:57-64), which is not fixed here.
make_tex_tables reads round-trip failures from eval_goldfish.json, which
measures every cell, rather than from manifest entries only a cell trained
with an --eval-texts slice ever had.
make_example_table.py generates the worked pre-tokenization example. It runs
the pretokenizers directly, before any vocabulary is learned, so it needs no
trained tokenizer and no corpus, and it reproduces the paper's table byte for
byte with the counts computed rather than typed.
Downstream
run_arms.sh is a plain sequential driver: it trains the matched tokenizers,
gates them, precomputes the byte factors, and then calls
run_downstream_eval.py once per arm x seed. Scheduling is left to whoever
reproduces it -- each model is one invocation of that script, and runs are
skip-if-done, so cells can be submitted anywhere and a re-run fills in the
rest. The CSCS Clariden job scripts, their site-specific environment and the
README describing that flow are deleted, as are the preflight and
corpus-prebuilding helpers that only existed to pack those jobs.
results_mingram.tsv takes the nine runs from #13, so the MinGram column has
no missing cell: bnd_w at 0.8763 and the case codes at 0.8760.
Deleted
The superseded FineWiki grids, probes and prototypes; 62 MB of FineWiki 32k
tokenizers; the root session logs; and the bits-per-byte-against-compression
figure, which is not in the paper. The 170 MB of matched tokenizers, the
generated .tex and the per-cell manifest fragments are untracked but kept
locally -- manifest.json holds exactly the fragments' union, so tracking both
committed the same 60 entries twice. What stays in git is 460 KB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7F6Ac72HbPG15ExjkbFFu
sanderland
added a commit
that referenced
this pull request
Aug 5, 2026
A reproduction directory in the shape of the other three papers, and one
pretokenizer configured by options instead of three classes.
./paper_utils/boundary/run_all_experiments.sh
regenerates every table from the caches committed under paper/generated/ -- no
GPU, no trained tokenizer, about a second. GRID=1 retrains the tokenizer grid,
DOWNSTREAM=1 reruns the LM sweep and exits with a usable message where there is
no CUDA.
The pretokenizer
BoundaryScriptPretokenizer is the only class. Case codes always sit outside
the span's markers and are applied only to a span that carries case; the
inside-the-markers placement and the version with no case test are gone.
Options replace the subclasses:
boundary_targets ("punct",) by default -- "word" is not a member, since
word spans are always delimited
shift_code <^> title case
caps_code <^^> all caps
min_caps_length shortest span <^^> may cover, 2
single_char_shift whether "A" is shift-coded, True
The defaults reproduce the reported arm exactly, so the committed caches keep
describing the default configuration. Verified before converting anything:
encode and decode agree with the old classes on 263 texts x 6 variants x 3
digit_handling settings, 0 mismatches. All 164 trained tokenizers embed their
config, so they were rewritten in place -- 84 converted, 56 deleted as cells
of the two removed schemes, 24 baselines untouched -- and each converted file
was checked by loading it and hash-matching the freshly built variant.
tests/test_boundary.py merges the two old suites and covers each option, 1412
tests.
The tables
table_intrinsic_main.tex and table_downstream_main.tex regenerate
byte-identical apart from the header naming the new script path.
bnd_wpd_extcapsfix is renamed bnd_wpd_caps across every cache. The
full-corpus cells are dropped: the paper reports the quick sample, the full
grid was 48 of a possible 108, and one cell (fineweb_ru_5gb_plain_mingram)
had landed at 34,684 against 34,685 everywhere else, which made
merge_manifests refuse to write. That cell is one token short because the
MinGram trainer tests the target outside its inner EM loop
(script_bpe/tokenizers/mingram/trainer.py:57-64), which is not fixed here.
make_tex_tables reads round-trip failures from eval_goldfish.json, which
measures every cell, rather than from manifest entries only a cell trained
with an --eval-texts slice ever had.
make_example_table.py generates the worked pre-tokenization example. It runs
the pretokenizers directly, before any vocabulary is learned, so it needs no
trained tokenizer and no corpus, and it reproduces the paper's table byte for
byte with the counts computed rather than typed.
Downstream
run_arms.sh is a plain sequential driver: it trains the matched tokenizers,
gates them, precomputes the byte factors, and then calls
run_downstream_eval.py once per arm x seed. Scheduling is left to whoever
reproduces it -- each model is one invocation of that script, and runs are
skip-if-done, so cells can be submitted anywhere and a re-run fills in the
rest. The CSCS Clariden job scripts, their site-specific environment and the
README describing that flow are deleted, as are the preflight and
corpus-prebuilding helpers that only existed to pack those jobs.
results_mingram.tsv takes the nine runs from #13, so the MinGram column has
no missing cell: bnd_w at 0.8763 and the case codes at 0.8760.
Deleted
The superseded FineWiki grids, probes and prototypes; 62 MB of FineWiki 32k
tokenizers; the root session logs; and the bits-per-byte-against-compression
figure, which is not in the paper. The 170 MB of matched tokenizers, the
generated .tex and the per-cell manifest fragments are untracked but kept
locally -- manifest.json holds exactly the fragments' union, so tracking both
committed the same 60 entries twice. What stays in git is 460 KB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7F6Ac72HbPG15ExjkbFFu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine runs, three arms by three seeds, jobs 2998893-95, every setting matched to the committed BPE and MinGram runs.
results_mingram.tsvgoes from 6 rows to 15, so the main table's MinGram column forbnd_wis filled in rather than blank.Cut from
6b6ba63and fast-forwards.bnd_wbnd_wpdOne number worth a look before the placement ablation stays dropped.
90cbf37took it out of the downstream tables, sobnd_wpd_capsandbnd_wpd_extcapsappear here only in the TSV. Paired by seed, positive meaning the second arm is lower:bnd_wpd_capsvsbnd_wpd_extcapsIndistinguishable under BPE, and under MinGram larger than the whole plain-to-
bnd_wpdeffect. There is a mechanism: with the code outside the markers the lower-cased span is a suffix of the cased form, so one word entry serves both, and MinGram prunes an overshot vocabulary with EM rather than growing greedily. Your commit adding the arm said the probe corpus could not answer this. Entirely your call whether it earns a place back.bnd_wat 0.876321 is close to extcaps but confounded: its byte factor is 0.8630 against plain's 1.0006, so it emits more tokens per byte.capsagainstextcapsis not, 0.8426 against 0.8444.Three seeds per arm. The t values are large because the seed spread is small.
Checked before reporting, since three earlier sweeps here produced plausible numbers that were wrong: the gain is in the raw bpb (1.0375 against 1.0444), not the byte-factor correction, whose two values differ by 0.21%; each arm has its own
token_bytes.pt; every run evaluated--step 2553, the checkpoint it had just trained; each tokenizer's sha256 matches what the in-job gate recorded; and the per-seed data permutations differ while being shared across arms, which is what makes the pairing valid. The tables regenerate byte for byte with your current generator.One change outside the results.
smoke_test.pytakes--corpus, so--require-matched-vocabcompares within one corpus and trainer. Without it the pattern matched all 75 MinGram tokenizers and demanded a single vocabulary across every corpus, sofineweb_ru_5gb_plain_mingramat 34,684 aborted all nine runs before any training. Verified both ways: the English arms pass, and asking forfineweb_ru_5gbstill refuses. That Russian cell is one token short because the MinGram trainer checks the target outside its inner EM loop (script_bpe/tokenizers/mingram/trainer.py:57-64), so pruning continues after the target is reached;manifest.jsonhere is merged with--no-checkand the check still reports it.