[mggarofalo]

Fixes #1102.

This makes chunking fast on documents with many break points and code fences, and chunk boundaries stay the same. A 1 MiB document of nothing but fence lines goes from about 4.4 s to about 0.1 s. A 10 MiB one goes from not finishing in 300 s to about 1.7 s. Ordinary documents gain little in absolute terms, because they were not slow to begin with.

The change

Both lookups now use binary search, through one small helper:

• findBestCutoff jumps to the first break point in its window. It used to walk every break point from the start of the document for each chunk.
• isInsideCodeFence looks only at the last fence that starts before the position. It used to scan every fence.

The second one relies on fences being sorted and not overlapping. findCodeFences guarantees that: each region ends no later than the next \n + ``` match starts, and regions are appended in order. The differential test below checks this on every input instead of assuming it. Neither function is exported from the package entry point, and the two doc comments now state what they require. Open PR #1074 calls isInsideCodeFence with findCodeFences output, so it stays compatible.

Chunks do not change

A change in chunk boundaries would quietly alter what gets embedded, so I tested this against the old build directly. A throwaway script loads the build of 86d2365 and the build of this branch in one process and compares their output.

The inputs were:

• 39 hand-written documents: unbalanced, nested, adjacent and tight fences, CRLF, CR only, mixed line endings, a BOM, NUL, zero-width characters, emoji and lone surrogates, and break points at, just before and just after the window edge and the target.
• 9 real files from this repository, with 2 of them also rewritten to CRLF.
• 13,000 seeded random documents, built from an alphabet weighted toward fences and line endings, with random chunk, overlap and window sizes (seeds 20261104 and 7).
• 2,000 runs per seed with extra break points merged in, as the AST path produces them, and 6 runs through chunkDocumentAsync with both strategies.

Compared	Count	Differences
Chunks, by position and text	27.6 million	0
findBestCutoff calls	10.0 million	0
isInsideCodeFence calls, at every position up to 20,000	104 million	0
Sorted and non-overlapping checks on break points and fences	Every document	0

To check that the script can fail, I ran it against 5 deliberately broken builds: an inclusive fence end, an inclusive fence start, a dropped window-edge break point, a skipped first break point, and an ignored first fence. It reported differences on all 5, between about 15,000 and 260,000 each.

I did not run the token-based path with a real tokenizer, because this work used no models. That path calls the same chunkDocument and chunkDocumentAsync compared above.

Before and after

Times are the median of 5 runs, with the fastest and slowest in brackets, unless noted. Each run used a fresh process, with builds interleaved. They cover the break point scan, the fence scan and chunking together, at default sizes.

Document	Size	Before	After
Only ``` lines	1 MiB	4.40 s (4.18 to 4.64)	0.11 s (0.11 to 0.14)
Only ``` lines	2 MiB	17.4 s (16.2 to 19.8)	0.20 s (0.18 to 0.28)
Only ``` lines	4 MiB	102 s (98 to 113), 3 runs	0.61 s (0.59 to 0.64), 3 runs
Only ``` lines	10 MiB	Not finished at 300 s, 1 run	1.7 s (1.6 to 1.8), 3 runs
Prose and short code blocks	10 MiB	2.42 s (2.32 to 2.61)	0.15 s (0.13 to 0.16)
Prose, no fences	10 MiB	0.62 s (0.60 to 0.69)	0.06 s (0.06 to 0.07)

The candidate loop in structuredSearch, with 40 candidates of 256 KiB and 3 runs each, goes from 5.3 s to 0.32 s on fence lines and from 103 ms to 67 ms on prose with short code blocks. The timings come from Node 24.18 on Windows 11 with a Ryzen 9 9900X, on a machine busy with other jobs. Issue #1102 has the full table and a reproduction script.

The scan is now the larger cost, and this PR does not change it. At 10 MiB of nothing but newlines, it takes about 5 s and 1.5 GiB of memory.

Tests

test/store.test.ts gains 8 tests:

• 2 bound how many elements each lookup reads among 100,000. Both fail on the old code, with about 100,000 reads against a limit of 100.
• 2 compare each function with a linear version on documents with awkward fences, at every position.
• 4 pin down edge cases: no fences, the edge shared by two tight fences, and break points exactly on the window edge and the target.

Results, on the same commit, for main and this branch:

Check	main	This branch
oxlint, Linux	0 errors	0 errors
tsc for the build and the whole project, Linux	Clean	Clean
vitest, Linux	1524 passed, 9 failed	1532 passed, 9 failed
bun test, Linux	1519 passed, 10 failed	1527 passed, 10 failed
vitest, Windows	1400 passed, 76 failed	1408 passed, 76 failed

The failing tests are the same on main and this branch in all 3 runs. I did not look into their causes, though I ran with QMD_FORCE_CPU=1 and no GPU, which may explain the QMD_LLAMA_GPU ones. The Linux runs used WSL with Node 24.18 and Bun 1.4.4, which differ from the versions in your CI. I did not run on macOS or on Node 22 or 23, and tests that need models were skipped.

I ran the code checks on the commit before the changelog entry, which is the only later change.

Not included

• A yield in the candidate loop. With this change and the existing body cap, the loop holds the event loop for about 0.3 s at worst and 30 to 70 ms otherwise, so I left it out. MCP HTTP server: initialize handshake blocks behind in-flight query under concurrency (single event loop, no worker offload) #762 covers the wider problem.
• The Map limit in scanBreakPoints. It throws above 2^24 newlines, which the 10 MB file limit keeps indexing from reaching.

#496 proposed the same binary searches, and its author closed it without a review. This PR adds the measurements at the sizes where the problem shows, the differential check, and tests.

Differential script

Usage: QMD_REPO=<checkout> node differential.mjs <old store.js> <new store.js> <fuzz iterations> [seed]

// Compares two builds of store.js on the same inputs. Any difference is a failure.
// usage: node differential.mjs <old store.js> <new store.js> <fuzz iterations> [seed]
import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";

const [oldPath, newPath, itersArg, seedArg] = process.argv.slice(2);
const A = await import(pathToFileURL(oldPath).href);
const B = await import(pathToFileURL(newPath).href);
if (A === B) throw new Error("same module loaded twice");
console.log("linear in old build:", /\.some\(/.test(A.isInsideCodeFence.toString()), "| linear in new build:", /\.some\(/.test(B.isInsideCodeFence.toString()));
if (A.isInsideCodeFence.toString() === B.isInsideCodeFence.toString()) throw new Error("the two builds have the same isInsideCodeFence: nothing is being compared");

let rng = Number(seedArg ?? 20261104) >>> 0;
const rand = () => { rng = (rng * 1664525 + 1013904223) >>> 0; return rng / 2 ** 32; };
const int = (lo, hi) => lo + Math.floor(rand() * (hi - lo + 1));
const pick = xs => xs[int(0, xs.length - 1)];

const stats = { docs: 0, chunkCalls: 0, cutoffCalls: 0, fenceCalls: 0, astCalls: 0, chunksCompared: 0, cutoffsThatFellInWindow: 0, posInsideFence: 0, posAtFenceEdge: 0 };
const failures = [];
const fail = (kind, detail) => { failures.push({ kind, ...detail }); if (failures.length <= 5) console.log("DIFF", kind, JSON.stringify(detail).slice(0, 400)); };
const same = (a, b) => a.length === b.length && a.every((x, i) => x.pos === b[i].pos && x.text === b[i].text);

function checkDoc(label, text, paramSets, targets) {
  stats.docs++;
  const bpA = A.scanBreakPoints(text), bpB = B.scanBreakPoints(text);
  const fA = A.findCodeFences(text), fB = B.findCodeFences(text);
  if (JSON.stringify(bpA) !== JSON.stringify(bpB) || JSON.stringify(fA) !== JSON.stringify(fB)) fail("scan", { label });

  // Invariants the binary searches rely on, checked rather than assumed.
  for (let i = 1; i < bpB.length; i++) if (bpB[i - 1].pos > bpB[i].pos) fail("bp-unsorted", { label, i });
  for (let i = 0; i < fB.length; i++) {
    if (fB[i].start > fB[i].end) fail("fence-inverted", { label, i });
    if (i && fB[i - 1].end > fB[i].start) fail("fence-overlap", { label, i });
  }

  // isInsideCodeFence at every position, plus just outside the document.
  const end = Math.min(text.length + 2, 20000);
  for (let p = -1; p < end; p++) {
    const a = A.isInsideCodeFence(p, fA), b = B.isInsideCodeFence(p, fB);
    stats.fenceCalls++;
    if (a) stats.posInsideFence++;
    if (a !== b) fail("isInside", { label, p, a, b });
  }
  for (const f of fB) if (f.start < end) stats.posAtFenceEdge += 2;

  // findBestCutoff directly, with windows that land exactly on break points.
  for (const t of targets(text, bpB)) {
    for (const w of [0, 1, 2, 3, 7, 50, 200, 800]) {
      for (const decay of [0, 0.7, 1]) {
        for (const fences of [fA, []]) {
          const a = A.findBestCutoff(bpA, t, w, decay, fences), b = B.findBestCutoff(bpB, t, w, decay, fences);
          stats.cutoffCalls++;
          if (a !== t) stats.cutoffsThatFellInWindow++;
          if (a !== b) fail("cutoff", { label, t, w, decay, a, b });
        }
      }
    }
  }

  // The whole chunker, with the defaults and with the sizes the token path uses.
  for (const [max, overlap, windowChars] of paramSets) {
    const ca = A.chunkDocument(text, max, overlap, windowChars), cb = B.chunkDocument(text, max, overlap, windowChars);
    stats.chunkCalls++;
    stats.chunksCompared += ca.length;
    if (!same(ca, cb)) fail("chunk", { label, max, overlap, windowChars, na: ca.length, nb: cb.length });
  }
}

const defaultParams = [[undefined, undefined, undefined], [2700, 405, 600], [1000, 150, 200], [200, 30, 60], [64, 8, 32], [16, 0, 8], [8, 7, 0], [5, 2, 100], [1, 0, 0]];
const edgeTargets = (text, bps) => {
  const ts = new Set([0, 1, text.length, text.length - 1, Math.floor(text.length / 2)]);
  for (const bp of bps.slice(0, 40)) for (const d of [-1, 0, 1, 7, 50, 51, 200, 800, 801]) { ts.add(bp.pos + d); }
  return [...ts].filter(t => t >= 0).slice(0, 120);
};

// --- Hand-written adversarial inputs ---
const f = "```";
const named = {
  empty: "",
  onlyFence: f,
  leadingFenceNoNewline: `${f}\ncode\n${f}\nafter\n\nmore`,
  unbalancedOne: `intro\n${f}\nnever closed\n\n# heading\n\nmore text\n\n## more\n`,
  unbalancedThree: `a\n${f}\nb\n${f}\nc\n${f}\nd\n\ne\n\nf`,
  tightEmpty: `\n${f}\n${f}\n${f}\n${f}\n${f}\n${f}\n`,
  tightNoNewlineAtEnd: `\n${f}\n${f}`,
  adjacentNoGap: `\n${f}${f}\n${f}${f}\n${f}`,
  longTicks: `x\n\`\`\`\`\ncode\n\`\`\`\ninner\n\`\`\`\`\ny\n\`\`\`\`\`\nz\n`,
  nestedFourInThree: `p\n${f}md\n\`\`\`\`\n${f}\nnested\n${f}\n\`\`\`\`\n${f}\nq\n`,
  tildes: `p\n~~~\ncode\n~~~\nq\n\n${f}\nx\n${f}\nz`,
  indented: `p\n  ${f}\ncode\n  ${f}\nq\n\t${f}\nr\n`,
  crlf: `intro\r\n\r\n${f}\r\ncode\r\n${f}\r\n\r\n# head\r\n\r\nbody\r\n---\r\nend\r\n`,
  crlfTight: `\r\n${f}\r\n${f}\r\n${f}\r\n${f}\r\n`,
  crOnly: `a\r${f}\rb\r${f}\rc\r\rd`,
  mixedEol: `a\n\r\n${f}\r\nb\n${f}\n\r\nc\r\r\n\n# h\n`,
  unicode: `é\n\n${f}\n😀😀😀\n${f}\n\n# 見出し\n\nçà et là — ünï\n\n${f}\n𝒳𝒴\n${f}\n\n- 😀\n1. é\n`,
  surrogateAtEdges: ("😀".repeat(40) + "\n\n").repeat(30),
  surrogateFences: (`\n${f}\n` + "😀".repeat(9)).repeat(40),
  bom: `${f}\ncode\n${f}\n\n# h\n`,
  zeroWidth: `a​\n${f}​\nb\n${f}\n
c
\n\nd`,
  nul: `a\0\n${f}\0\nb\0\n${f}\n\0\n\nc`,
  headingsInFence: `# a\n${f}\n# not a heading\n## nor this\n\n\n${f}\n# b\n`,
  hrs: `a\n---\nb\n***\nc\n___\nd\n--- \n\n***\t\n`,
  blankRuns: `a${"\n".repeat(50)}b${"\n".repeat(3)}${f}${"\n".repeat(20)}${f}${"\n".repeat(9)}c`,
  onlyNewlines: "\n".repeat(5000),
  noBreaks: "x".repeat(10000),
  fenceAtWindowEdge: "x".repeat(3599 - 800 - 1) + `\n${f}\n` + "y".repeat(800) + `\n${f}\n` + "z".repeat(4000),
  headingAtExactEdge: "x".repeat(3600 - 800 - 1) + "\n# edge\n" + "y".repeat(5000),
  headingOneBeforeEdge: "x".repeat(3600 - 800 - 2) + "\n# edge\n" + "y".repeat(5000),
  breakAtTarget: "x".repeat(3599) + "\n# target\n" + "y".repeat(5000),
  breakAfterTarget: "x".repeat(3600) + "\n# target\n" + "y".repeat(5000),
  fenceCloseAtTarget: "x".repeat(3596) + `\n${f}\n` + "y".repeat(100) + `\n${f}\n` + "z".repeat(5000),
  backticks64k: `${f}\n`.repeat(16384),
  backticksOdd: `${f}\n`.repeat(16385),
  backticksWithLeadingNewline: `\n${f}`.repeat(16384),
  manyBlocks: (`para\n\n${f}ts\nconst a=1;\n${f}\n\n`).repeat(3000),
  manyHeadings: (`## h\ntext\n`).repeat(6000),
  lists: (`- a\n1. b\n* c\n`).repeat(4000),
};
const tsDoc = Array.from({ length: 200 }, (_, i) => `export function fn${i}(a: number) {\n  // \`\`\` not a fence\n  return a + ${i};\n}\n`).join("\n");
for (const [label, text] of Object.entries(named)) checkDoc(label, text, defaultParams, edgeTargets);
console.log("named docs done", stats.docs, "failures", failures.length);

// --- Files in the repository itself (real prose, code, and fences) ---
const repo = process.env.QMD_REPO;
if (repo) {
  const paths = ["README.md", "CHANGELOG.md", "CLAUDE.md", "DESIGN.md", "src/store.ts", "src/llm.ts", "src/cli/qmd.ts", "test/store.test.ts", ...readdirSync(join(repo, "docs")).filter(n => n.endsWith(".md")).map(n => `docs/${n}`)];
  for (const p of paths) checkDoc(`repo:${p}`, readFileSync(join(repo, p), "utf8"), defaultParams, edgeTargets);
  // Line endings rewritten to CRLF too.
  for (const p of ["README.md", "CHANGELOG.md"]) checkDoc(`repo-crlf:${p}`, readFileSync(join(repo, p), "utf8").replace(/\r?\n/g, "\r\n"), defaultParams, edgeTargets);
  console.log("repo docs done", stats.docs, "failures", failures.length);
}

// --- Seeded fuzz from an alphabet weighted toward things that break chunkers ---
const alphabet = ["\n", "\n", "\n\n", "\r\n", "\r", "```", "```", "\n```", "\n```\n", "\n```ts\n", "````", "~~~", "`", "``", "# ", "\n# ", "\n## ", "\n### ", "\n---\n", "\n***\n", "\n- ", "\n1. ", "a", "word ", "é", "😀", "\uD83D", "\uDE00", "\0", "
", "\t", "    "];
const iters = Number(itersArg);
for (let i = 0; i < iters; i++) {
  const pieces = int(0, 3) === 0 ? int(300, 1500) : int(0, 300);
  const weight = int(1, 6);  // Some documents are mostly filler, some mostly markup.
  let text = "";
  for (let p = 0; p < pieces; p++) text += rand() < 0.15 * weight ? "x".repeat(int(1, 120)) : pick(alphabet);
  const max = int(1, 400), params = [
    [undefined, undefined, undefined],
    [max, int(0, max), int(0, 500)],
    [int(1, 60), 0, int(0, 40)],
    [int(10, 2000), int(0, 200), int(0, 800)],
  ];
  const targets = (t, bps) => {
    const ts = new Set();
    for (let k = 0; k < 8; k++) ts.add(int(0, t.length + 5));
    for (let k = 0; k < 8 && bps.length; k++) { const bp = pick(bps); ts.add(Math.max(0, bp.pos + pick([0, 1, 50, 200, 800]))); }
    return [...ts];
  };
  checkDoc(`fuzz:${i}`, text, params, targets);
}
console.log("fuzz done", stats.docs, "failures", failures.length);

// --- Merged break points, as the AST path produces them ---
for (let i = 0; i < 2000; i++) {
  const text = Array.from({ length: int(5, 200) }, () => pick(alphabet)).join("") + "x".repeat(int(0, 3000));
  const extra = Array.from({ length: int(0, 60) }, () => ({ pos: int(0, text.length), score: pick([1, 20, 60, 90, 100]), type: "ast" }));
  const merged = A.mergeBreakPoints(A.scanBreakPoints(text), extra);
  const fences = A.findCodeFences(text);
  const max = int(5, 800), o = int(0, Math.floor(max / 2)), w = int(0, 300);
  const a = A.chunkDocumentWithBreakPoints(text, merged, fences, max, o, w), b = B.chunkDocumentWithBreakPoints(text, merged, fences, max, o, w);
  stats.chunkCalls++; stats.chunksCompared += a.length;
  if (!same(a, b)) fail("merged", { i, max, o, w });
}

// --- The real AST path ---
for (const [content, name] of [[tsDoc, "a.ts"], [named.manyBlocks, "a.md"], [tsDoc.replace(/\n/g, "\r\n"), "b.ts"]]) {
  for (const strategy of ["regex", "auto"]) {
    const a = await A.chunkDocumentAsync(content, undefined, undefined, undefined, name, strategy);
    const b = await B.chunkDocumentAsync(content, undefined, undefined, undefined, name, strategy);
    stats.astCalls++; stats.chunksCompared += a.length;
    if (!same(a, b)) fail("async", { name, strategy, na: a.length, nb: b.length });
  }
}

console.log(JSON.stringify({ stats, failures: failures.length, seed: seedArg ?? 20261104 }));
process.exit(failures.length ? 1 : 0);

An AI agent (Claude Code) wrote this change, ran these measurements and wrote this description on behalf of the account owner.

🤖 Generated with Claude Code