Skip to content

Put the MinGram and extcaps runs into the downstream tables - #12

Merged
sanderland merged 1 commit into
sanderland:claude/fineweb-space-neighbors-k10ufwfrom
cimeister:claude/downstream-tables
Aug 4, 2026
Merged

Put the MinGram and extcaps runs into the downstream tables#12
sanderland merged 1 commit into
sanderland:claude/fineweb-space-neighbors-k10ufwfrom
cimeister:claude/downstream-tables

Conversation

@sanderland

Copy link
Copy Markdown
Owner

Both sweeps were committed as TSVs and neither reached a table, so a reader of the paper saw the BPE arms only. Three things kept them out.

ARM_ORDER did not list bnd_wpd_extcaps, and the generator emits a row only for arms in that list, so pointing it at the extcaps TSV would have produced nothing.

The generator read a single --results file. The extcaps sweep ran under its own OUT and so has its own TSV, although its runs are trainer bpe like the rest of the main table. --results now takes a comma-separated list and defaults to the main sweep plus the extcaps sweep. Loading asserts one trainer across everything read, because arm is the grouping key and does not carry the trainer, so a mixed load would average a BPE and a MinGram tokenizer into one mean. Verified: passing results.tsv and results_mingram.tsv together is refused.

MinGram needs its own table for that same reason, so main() takes --trainer, which also selects the manifest rows for the compression half. downstream_tables_mingram.tex is the new output. --no-appendix skips the seed and shard tables, which cover the BPE sweep only.

Labels and captions are now per trainer. Both files previously emitted \label{tab:downstream-lm} and \label{tab:downstream-compression}, so LaTeX would resolve every \Cref to whichever file it read last, without an error. The captions also now name the trainer, which is the only thing distinguishing the two tables at a glance.

manifest.json is regenerated with --no-check, so the compression half can see the extcaps cell. The check reports
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685, 'plain': 34684}, the one cell the MinGram trainer pruned a token past its target (script_bpe/tokenizers/mingram/trainer.py:57-64 checks the size outside the inner EM loop). That refusal is accurate and the default stays on; it is overridden once, here, because the alternative is leaving a measured arm out of the paper. 152 entries, none removed, none altered.

Rows added, all matching values computed independently from the TSVs:

BPE downstream    bnd_wpd_extcaps  0.8793 +/- 0.0005 over 3 seeds
BPE compression   bnd_wpd_extcaps  3.7464 chars/token, +3.04% against plain
MinGram downstream  plain    0.8837 +/- 0.0008
                    bnd_wpd  0.8805 +/- 0.0006
MinGram compression plain 3.7077, bnd_w 3.1114, bnd_wp 3.5446, bnd_wpd 3.7713

Regenerating twice reproduces both files byte for byte.

Both sweeps were committed as TSVs and neither reached a table, so a reader of the paper
saw the BPE arms only. Three things kept them out.

ARM_ORDER did not list bnd_wpd_extcaps, and the generator emits a row only for arms in
that list, so pointing it at the extcaps TSV would have produced nothing.

The generator read a single --results file. The extcaps sweep ran under its own OUT and
so has its own TSV, although its runs are trainer bpe like the rest of the main table.
--results now takes a comma-separated list and defaults to the main sweep plus the
extcaps sweep. Loading asserts one trainer across everything read, because `arm` is the
grouping key and does not carry the trainer, so a mixed load would average a BPE and a
MinGram tokenizer into one mean. Verified: passing results.tsv and results_mingram.tsv
together is refused.

MinGram needs its own table for that same reason, so main() takes --trainer, which also
selects the manifest rows for the compression half. downstream_tables_mingram.tex is the
new output. --no-appendix skips the seed and shard tables, which cover the BPE sweep only.

Labels and captions are now per trainer. Both files previously emitted
\label{tab:downstream-lm} and \label{tab:downstream-compression}, so LaTeX would resolve
every \Cref to whichever file it read last, without an error. The captions also now name
the trainer, which is the only thing distinguishing the two tables at a glance.

manifest.json is regenerated with --no-check, so the compression half can see the extcaps
cell. The check reports
mingram on fineweb_ru_5gb: {'bnd_w': 34685, 'bnd_wp': 34685, 'bnd_wpd': 34685,
'plain': 34684}, the one cell the MinGram trainer pruned a token past its target
(script_bpe/tokenizers/mingram/trainer.py:57-64 checks the size outside the inner EM
loop). That refusal is accurate and the default stays on; it is overridden once, here,
because the alternative is leaving a measured arm out of the paper. 152 entries, none
removed, none altered.

Rows added, all matching values computed independently from the TSVs:

    BPE downstream    bnd_wpd_extcaps  0.8793 +/- 0.0005 over 3 seeds
    BPE compression   bnd_wpd_extcaps  3.7464 chars/token, +3.04% against plain
    MinGram downstream  plain    0.8837 +/- 0.0008
                        bnd_wpd  0.8805 +/- 0.0006
    MinGram compression plain 3.7077, bnd_w 3.1114, bnd_wp 3.5446, bnd_wpd 3.7713

Regenerating twice reproduces both files byte for byte.
@sanderland
sanderland merged commit d66b1e0 into sanderland:claude/fineweb-space-neighbors-k10ufw Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants