Claude/multilang grid - #10
Conversation
…and en:bnd_wp
Completes the 5-language by 5-arm grid at total vocabulary 34,685 on FineWeb 5 GB. With
the four English arms already present this makes 25 cells, every one at vocabulary 34,685
with 0 round-trip failures, read from the manifest fragments rather than asserted.
Jobs 2989992-95 on four nodes, five or six cells concurrently per node at 46 workers each,
all COMPLETED with fail=0 in 2h54m to 3h20m. Each cell was invoked as
train_multilang.py --lang <lang> --arms <arm> --trainers bpe --workers 46 --no-commit
--trainers bpe is passed explicitly, because this branch predates the change that made
both trainers the default; a rerun without the flag would not reproduce these cells.
--no-commit because train_multilang.py's concurrency safety is unique filenames per cell,
which protects several machines each with a clone but not several jobs sharing one
checkout, as on a cluster.
cluster/pack_cells.sbatch runs several cells concurrently on one node. Every node on this
account carries 4 GPUs and a node-hour bills as 4 GPU-hours whether or not a GPU is
touched, so the first layout, one cell per node across 26 nodes, spent 104 GPU-hours an
hour to run work that uses no GPU. Packing pays here only because a single cell cannot
saturate 288 cores: corpus building alternates parallel encode with a serial merge in the
parent, measured at 91% of one core with every worker idle.
These cells use the standard sampler, so their corpus names carry no _quick suffix and
none of the filenames collide with the _quick cells already on this branch.
plain and bnd_wpd repeated with the MinGram trainer, three seeds, every other setting
matched to the committed BPE runs. Two arms rather than four because plain against
bnd_wpd is the comparison that carries the claim: bnd_wpd emits fewer tokens per byte
than plain, so its lower loss cannot come from spending more forward passes per byte.
Validation bits per true byte, seeds 0 to 2, mean and sample standard deviation, with the
BPE runs restricted to the same three seeds:
trainer plain bnd_wpd
bpe 0.885315 (0.000312) 0.879999 (0.000487)
mingram 0.883694 (0.000808) 0.880507 (0.000625)
Paired by seed, plain minus bnd_wpd, positive meaning bnd_wpd is lower:
bpe +0.005317 SE 0.000319 t(2) = 16.7
mingram +0.003187 SE 0.000209 t(2) = 15.2
The direction holds under MinGram and the size is 60% of the BPE difference. Both sweeps
are paired the same way, one training-data-order permutation per seed shared across arms.
MinGram does not take an existing vocabulary. It trains its own BPE at
additional_vocab_size * overshoot_factor and prunes down with EM, so these are full
retrains. Because it prunes down to a target it can stop short, which BPE cannot, so
mingram_preflight.py gates on exact vocabulary 34,685 before any GPU time, along with
round-trip, chars/token within 8% of the BPE tokenizer for the same arm, that only BOS
carries zero bytes, and the CORE prefix property.
CORE_SAFE_ARMS is measured, not inherited. run_arms.sh's default was measured on BPE
tokenizers, and MinGram segments with a dynamic program rather than greedy merge replay,
so an arm CORE-safe under BPE need not be under MinGram. submit_mingram.sh now runs
core_prefix_check.py and refuses to submit unless the value passed matches the
measurement: plain 0/512 aborts, bnd_wpd 399/512, so plain is scored core,bpb and
bnd_wpd bpb, exactly as in the BPE sweep. Getting this wrong costs an hour of training
per run and leaves a log with no artifact_dir, which every later invocation resubmits.
cluster/pack_runs.sbatch runs up to 4 downstream runs on one node, one GPU each through
CUDA_VISIBLE_DEVICES, splitting the cores through a new ENCODE_WORKERS variable that
becomes run_downstream_eval.py --encode-workers. That flag reaches the encode pool only,
and Pool.map returns in input order, so it changes throughput and nothing the model sees;
MinGram at 70 workers is comparable to BPE at 287. The runs are GPU-bound, contrary to
the comment that justified giving each run a whole node: plain and bnd_wpd both run 2553
steps at bf16_mfu 34.4% and 27.5%, per-step time rises and falls with MFU, and no log
reports a dataloader stall. Six runs cost 16 GPU-hours rather than about 48.
run_arms.sh also changes CORE_SAFE_ARMS from ${X:-default} to ${X-default}. A sweep where
no arm satisfies the prefix property passes an explicit empty string, and the old form
substituted the default and scored those arms on CORE anyway.
collect_results.py refuses a log directory that mixes trainers. `arm` is the grouping key
everywhere downstream and does not carry the trainer, so one TSV holding both would
average a BPE and a MinGram tokenizer into a single mean per arm. For the same reason
these rows are in results_mingram.tsv rather than results.tsv.
…ment manifest.json on this branch names four tokenizer files under downstream/tokenizers that are not in the repository, so the committed English downstream result rests on artifacts nobody else can obtain. The project rule is that every number traces to a run, a config and a code version; this closes that gap. They are 1.17 MB each, the same size as the 27 tokenizers already tracked here. merge_manifests.py now folds in every committed fragment rather than the English four, so manifest.json goes from 4 entries to 54: 27 from this branch's existing cells and 27 from the grid. It also applies the cross-arm assertion that a single-process train_matched.py run does at the end, that every arm of one trainer and corpus lands on the same total vocabulary. The existing cells carry no eval_chars_per_token or roundtrip_failures because they were trained without --eval-texts. Their entries are merged as they are rather than filled in. One consequence in the generated table: make_tex_tables.py filters the manifest to fineweb_en_5gb and bpe, which now finds five arms instead of four, so the compression table gains the bnd_wp row it was previously missing.
3ceacee to
684e485
Compare
|
Rebased onto The branch was 47 commits behind, and I rewrote it rather than merging, so the 10 commits are replaced by 3. What changed and why: The conflict was one file, Two files I did not bring across, because this branch is older than yours. Dropped as out of scope for a results PR: the Slurm account changes across six scripts (specific to which CSCS budget we bill, no bearing on any number), and the One thing you may not expect, in the third commit. No filename collides. Your cells are Still to come, not in this PR: intrinsic vocabulary metrics for the new tokenizers, and |
8967364
into
sanderland:claude/fineweb-space-neighbors-k10ufw
2 participants
No description provided.