Boundary-marker pretokenization

[sanderland]

Adds the pretokenizer and reproduction directory for Explicit Boundary Markers for Subword Vocabularies.

The scheme

A single atomic token <|> delimits spans, and the space between two delimited spans is elided at encode time and reconstructed at decode time from the resulting pair of touching markers:

two <|> touching  ->  exactly one space
a lone <|>        ->  nothing (structural boundary only)

That removes the duplication the leading-space convention creates, where ' the' and 'the' are separate vocabulary entries. Word spans are delimited unconditionally, so a word has one canonical form regardless of what precedes it; punctuation and digits are delimited only on a side whose space was elided, which is what keeps the touching-marker signal unambiguous.

Optional case codes go a step further: a title-case span is emitted as <^> plus its lowercased form, an all-caps span as <^^>, so The/the and NASA/nasa can share an entry. The code sits outside the span's markers, which is what makes sharing possible — <|>the<|> occurs inside <^><|>the<|> as a suffix, so the trainer may cover both with one piece, and will spend a second entry only where the frequency justifies it.

script_bpe/pretokenize's existing schemes are untouched; this is a new ScriptPretokenizer subclass in paper_utils/boundary/boundary_pretokenizer.py, configured by options:

option
boundary_targets	what is delimited besides word spans, which always are: (), ("punct",), ("punct","digit")
shift_code / caps_code	<^> title case, <^^> all caps
min_caps_length	shortest span <^^> may cover
single_char_shift	whether a one-character span is shift-coded

It is a SCRIPT-v3 scheme and the module docstring says so: span identification reads v3's category folding, so V1/V2 would silently mark nothing, and the byte-based pretokenizers are out of reach. Making it universal is scoped out and described.

Reproducing the paper

./paper_utils/boundary/run_all_experiments.sh

regenerates every table from the measurement caches committed under paper/generated/ — no GPU, no trained tokenizer, about a second, because those caches are tracked precisely so the paper reproduces on any machine. GRID=1 retrains the six-language tokenizer grid; DOWNSTREAM=1 reruns the LM sweep and exits with a usable message where there is no CUDA.

Produces the two main tables (compression + MorphScore; bits per byte), their appendix companions, a worked pre-tokenization example, and a table of duplicate vocabulary entries. Every number is computed from an artifact rather than transcribed, including the ones quoted in captions.

The downstream sweep is a plain sequential driver — one invocation of run_downstream_eval.py per arm × seed, skip-if-done — so scheduling it across nodes is left to whoever reproduces it rather than baked in.

Shared code

• script_bpe/corpus/registry.py: a sampled-text cache keyed on the sampling parameters, so a grid scans the source once instead of once per arm; and create_streaming_quick_corpus, which reads until its character budget is full instead of reservoir-sampling the whole source. Uniform sampling of a 5 GB slice needs ~45 GB read per language; the quick sampler is a deliberately separate corpus name so it can never share a cache or a tokenizer path with a full-sample build.
• script_bpe/utils.py, corpus/base.py, bpe/trainer.py: bounded worker shutdown. The forkserver can leave an exited worker unreaped, and the parent then blocks forever in join() on a sentinel that never fires — twice costing 8.6 hours on a corpus build whose results the parent already held. join_workers/shutdown_pool bound the wait and escalate. Also fixes a Counter.__iadd__ rescan that made chunk-count merging quadratic.
• eval/py-nanochat: bits-per-byte per true byte. nanochat divides summed loss by the summed byte length of the target tokens, taken from each token's own decoding; a scheme that elides a character between two tokens has no token to charge it to, so the denominator is short and bpb is inflated in proportion to how much the scheme elides. byte_factor measures the discrepancy over the same text and makes the figure comparable across tokenizers.
• eval/py-nanochat: run_experiment(shuffle_data_order=...), off by default. nanochat's document iterator holds no RNG, so every seed of a configuration reads the identical document sequence and a spread over seeds measures weight initialization alone. The option lets the seed permute the training document order too; the permutation is derived from the seed and not the tokenizer, so arms of a comparison stay paired. It defaults off because it changes what a given seed means, and existing measurements would not reproduce with it on.

Tests

tests/test_boundary.py (642 cases): round-trip, the touching-marker/space invariant, no-triple-marker, cross-script span merging, digit handling under SPLIT/RTL3, non-ASCII numerics, hash distinctness across every axis that keys the corpus cache, and each case option gated independently. Plus test_quick_sample.py and test_sample_cache.py for the sampler and its cache.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7F6Ac72HbPG15ExjkbFFu