Boundary-marker pretokenization: experiments, paper draft, and downstream LM setup

[sanderland]

A single atomic token <|> delimits spans, and the space between two adjacent delimited spans is elided at encode time and reconstructed from the resulting pair of touching markers. This removes the ' the'/'the' duplication the leading-space convention creates.

Design

Word spans are delimited unconditionally on both sides, so a word has one canonical form regardless of what precedes it. A word span is a maximal run from any space-using script, merged across script changes — without merging, two unconditional markers meet at a script boundary, become indistinguishable from an elided space, and decode fabricates one. Punctuation and digits are delimited only on a side whose single space was elided, which is required for the same reason.

bpe_merge_allowed bans merges across touching markers, so '<|>the<|><|>' can never become a token.

Results

1 GB FineWiki, 6 languages × 4 pretokenizers × {BPE, MinGram}, 0 round-trip failures. Boundary markers on words + punctuation + digits (bnd_wpd) beat the SCRIPT-v3 baseline in all six languages, mean +2.14 % chars/token (en +3.77, de +1.60, fi +2.02, ru +2.67, ar +1.91, ko +0.88). Delimiting words alone is much worse (−13.75 %) — punctuation and digits are what make the scheme pay. Duplicate ' X'/'X' pairs drop from 3,159–5,244 to 4–10; the baseline spends 18.3–30.4 % of its vocabulary on them.

MinGram (250 M) preserves the ordering, and Korean changes sign with scale (+0.88 % at 1 GB, −0.67 % at 250 M).

Digit handling bounds the markable set. Unbounded, every distinct number acquires up to four marked forms; RTL3 caps it at 1,110 strings and SPLIT at 10. Measured on English: None +3.38 %, SPLIT +3.71 %, RTL3 +3.37 % — the tax is real but worth only +0.33 pp.

Caps codes (Claude-3 style <^>/<^^> on whole word spans) cut case-duplicate pairs from 5,017 to 91, i.e. 29.1 % → 0.5 % of the vocabulary, for −0.04 % compression. Which is the recurring finding, now three times over: reclaiming vocabulary does not convert into compression.

Downstream LM setup

The compression numbers say nothing about modelling quality, which was the draft's principal limitation. marker_experiments/downstream/ wires the tokenizers into the existing pynanochat harness — the same path the MinGram downstream table used — and adds the four things that path needed:

• boundary_tokenizer.py — Pretokenizer.REGISTRY is filled by __init_subclass__, so a fresh subprocess calling BPETokenizer.load on a boundary model raises KeyError: 'BoundaryScriptPretokenizer'. This module imports for the registration side effect and re-exports the tokenizer classes unchanged (BoundaryBPETokenizer is BPETokenizer), making the fix one --tokenizer-class flag rather than a subclass.
• train_matched.py — the grid fixed additional_vocab_size=32768, so the arms have different total vocabularies (34478/34479/34481, since the marker and caps codes are extra atomic tokens). Right for a compression control; wrong downstream, where vocabulary size sets the parameter count and hence nanochat's token horizon. This matches the total instead (default 34,685, the MinGram table's) and exits non-zero if the arms disagree.
• smoke_test.py — fresh-process load for both serialisations, the pynanochat.Tokenizer contract, dense-id space and synthetic BOS, the vocab <= 65535 bound, adapter round-trip on marker/caps/digit/mixed-script text, batch-vs-single encode.
• run_arms.sh + collect_results.py — resumable train → check → run → collect, producing a TSV whose columns match what the paper's table generators already read.

README.md there has the full cluster instructions: branch, editable-install requirement, a no-GPU verification step, SMOKE=1 pipeline check, the knob table, a Slurm array template, and what to send back.

Testing

• marker_experiments/test_boundary.py: 599 passed — round-trip (including tests/data/taylorswift.txt), touching-pair and no-triple-marker invariants, span merging across scripts, per-target behaviour, multi-space runs left untouched, the merge constraint, config-hash distinctness, digit_handling, non-ASCII numerics, caps codes.
• smoke_test.py: 0 failures over all 56 checked-in tokenizers and over a freshly trained matched set.
• The three original boundary variants keep their config hashes, so cached corpora stay valid.

The GPU leg is untested. This container has neither torch nor a GPU, so write_token_bytes and everything downstream of it are unexercised. SMOKE=1 covers exactly that leg, and the README states this plainly.

Three caveats recorded in the work

• Evaluation leak. The held-out slice was initially also in the training corpus. train_batches() now withholds it, but the committed 1 GB numbers predate that fix: the leak is identical across pretokenizers, so the reported gaps are unaffected while absolute chars/token is optimistic. Disclosed in §4.1 of the draft.
• A filename collision found while smoke-testing. caps_grid.py (250 M chars) wrote the same paths as the 1 GB grid and had silently replaced that grid's en_bnd_wpd artifact — three "different" tokenizers produced byte-identical merge lists, which is what surfaced it. Its outputs are now prefixed and the two affected files renamed to what they actually are. The 1 GB results are unaffected — that cell recorded unique_chunks=2,072,665 against 906,491 for the 250 M corpus — and no downstream arm reuses grid artifacts. The 1 GB en_bnd_wpd artifact itself is gone and would need retraining.
• A second unregistered pretokenizer. en_plain_{None,SPLIT,RTL3} also failed fresh-process load, with KeyError: 'DigitAwareScriptPretokenizer' — same root cause, different class, defined inside digit_split_grid.py. Moved to marker_experiments/digit_pretokenizer.py; the registry is keyed by class name, so the config hash is unchanged (PT-ec369df5, still matching the cached digitsplit250_en_plain_SPLIT corpus) and no result changes.

Also included: marker_experiments/paper.md and the ACL LaTeX port under marker_experiments/paper/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PE4L4w3oY91vHCMK9uw32k