Put the MinGram and extcaps runs into the downstream tables - #12
Merged
sanderland merged 1 commit intoAug 4, 2026
Conversation
Both sweeps were committed as TSVs and neither reached a table, so a reader of the paper
saw the BPE arms only. Three things kept them out.
ARM_ORDER did not list bnd_wpd_extcaps, and the generator emits a row only for arms in
that list, so pointing it at the extcaps TSV would have produced nothing.
The generator read a single --results file. The extcaps sweep ran under its own OUT and
so has its own TSV, although its runs are trainer bpe like the rest of the main table.
--results now takes a comma-separated list and defaults to the main sweep plus the
extcaps sweep. Loading asserts one trainer across everything read, because `arm` is the
grouping key and does not carry the trainer, so a mixed load would average a BPE and a
MinGram tokenizer into one mean. Verified: passing results.tsv and results_mingram.tsv
together is refused.
MinGram needs its own table for that same reason, so main() takes --trainer, which also
selects the manifest rows for the compression half. downstream_tables_mingram.tex is the
new output. --no-appendix skips the seed and shard tables, which cover the BPE sweep only.
Labels and captions are now per trainer. Both files previously emitted
\label{tab:downstream-lm} and \label{tab:downstream-compression}, so LaTeX would resolve
every \Cref to whichever file it read last, without an error. The captions also now name
the trainer, which is the only thing distinguishing the two tables at a glance.
manifest.json is regenerated with --no-check, so the compression half can see the extcaps
cell. The check reports
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685,
'plain': 34684}, the one cell the MinGram trainer pruned a token past its target
(script_bpe/tokenizers/mingram/trainer.py:57-64 checks the size outside the inner EM
loop). That refusal is accurate and the default stays on; it is overridden once, here,
because the alternative is leaving a measured arm out of the paper. 152 entries, none
removed, none altered.
Rows added, all matching values computed independently from the TSVs:
BPE downstream bnd_wpd_extcaps 0.8793 +/- 0.0005 over 3 seeds
BPE compression bnd_wpd_extcaps 3.7464 chars/token, +3.04% against plain
MinGram downstream plain 0.8837 +/- 0.0008
bnd_wpd 0.8805 +/- 0.0006
MinGram compression plain 3.7077, bnd_w 3.1114, bnd_wp 3.5446, bnd_wpd 3.7713
Regenerating twice reproduces both files byte for byte.
sanderland
merged commit Aug 4, 2026
d66b1e0
into
sanderland:claude/fineweb-space-neighbors-k10ufw
1 check passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both sweeps were committed as TSVs and neither reached a table, so a reader of the paper saw the BPE arms only. Three things kept them out.
ARM_ORDER did not list bnd_wpd_extcaps, and the generator emits a row only for arms in that list, so pointing it at the extcaps TSV would have produced nothing.
The generator read a single --results file. The extcaps sweep ran under its own OUT and so has its own TSV, although its runs are trainer bpe like the rest of the main table. --results now takes a comma-separated list and defaults to the main sweep plus the extcaps sweep. Loading asserts one trainer across everything read, because
armis the grouping key and does not carry the trainer, so a mixed load would average a BPE and a MinGram tokenizer into one mean. Verified: passing results.tsv and results_mingram.tsv together is refused.MinGram needs its own table for that same reason, so main() takes --trainer, which also selects the manifest rows for the compression half. downstream_tables_mingram.tex is the new output. --no-appendix skips the seed and shard tables, which cover the BPE sweep only.
Labels and captions are now per trainer. Both files previously emitted \label{tab:downstream-lm} and \label{tab:downstream-compression}, so LaTeX would resolve every \Cref to whichever file it read last, without an error. The captions also now name the trainer, which is the only thing distinguishing the two tables at a glance.
manifest.json is regenerated with --no-check, so the compression half can see the extcaps cell. The check reports
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685, 'plain': 34684}, the one cell the MinGram trainer pruned a token past its target (script_bpe/tokenizers/mingram/trainer.py:57-64 checks the size outside the inner EM loop). That refusal is accurate and the default stays on; it is overridden once, here, because the alternative is leaving a measured arm out of the paper. 152 entries, none removed, none altered.
Rows added, all matching values computed independently from the TSVs:
Regenerating twice reproduces both files byte for byte.