Skip to content

fix(CommandPalette): cut search highlights on grapheme clusters, not code points - #371

Merged
IgorShevchik merged 1 commit into
mainfrom
claude/search-grapheme-clusters
Aug 13, 2026
Merged

fix(CommandPalette): cut search highlights on grapheme clusters, not code points#371
IgorShevchik merged 1 commit into
mainfrom
claude/search-grapheme-clusters

Conversation

@IgorShevchik

@IgorShevchik IgorShevchik commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Linked issue

Closes #364.

Now targets main#369 has landed (531921ad), so this is a single commit and CI runs on it for the first time. #362 and #375 are covered by @arsalan507's #379, not here — the overlapping work was dropped. This PR will need a rebase once #379 lands, since the grapheme snap subsumes its surrogate snap.

Authorship — please read before reviewing. This branch, its three review rounds and every revision were produced by a single Claude Code session. Nothing here has had an independent reader. The measurements are reproducible but unaudited, and self-review does not substitute for review: each round found real defects in what the previous round had signed off, which is evidence that the process is not converging on its own. This is not only about the v-html-adjacent parts — the correctness argument, the benchmark methodology and the invariant text all carry the same caveat.

Type of change

  • Documentation (updates to the documentation or readme)
  • Bug fix (a non-breaking change that fixes an issue)
  • Enhancement (improving an existing functionality)
  • New feature (a non-breaking change that adds functionality)
  • Chore (updates to the build process or auxiliary tools and libraries)
  • Breaking change (fix or feature that would cause existing functionality to change)

Description

Code points are not what a reader sees. A flag is two regional indicators, a family emoji several joined by ZWJ, कि a consonant plus a vowel sign. Cutting inside one produces a different character rather than a broken one — 🇺🇸 truncated by a single code point re-pairs into 🇸🇺, a different country, with nothing to signal the loss. That is worse than a , which at least looks wrong.

Both boundaries snap off the straddled cluster: mark insertion in generateHighlightedText, truncation in truncateHTMLFromStart.

Chosen on measurement.

approach cost correct?
segment the whole string 455 µs at 979 chars, 52 ms at 100k yes, unusable
fixed ±64 window constant 3.2 µs no
Segments.containing() see below yes

The window is worth knowing about, because it is the obvious optimisation and looks ideal. A run of flags is longer than the window, so segmentation starts mid-run and re-pairs the indicators — reproducing the exact bug being fixed, 28 disagreements in 1044 probes.

So containing() ships, behind three guards:

  • A < U+0300 screen, keeping ASCII and Latin-1 off the segmenter. It does nothing for Cyrillic (U+0430) or CJK — both above the floor. For a product localised into Russian this covers markup and Latin identifiers, not body text. perf(search): the grapheme-snap fast path skips Cyrillic and CJK — every boundary reaches the segmenter #376 tracks widening it.
  • One segmenter view per value, built lazily: 8.1 ms → 0.6 ms over 1600 boundaries.
  • An 8192-character ceiling, past which only the surrogate snap applies and every multi-code-point cluster loses protection — not just flags. Note it measures the marked-up string, so the threshold arrives 13 characters earlier than a reading of value.length suggests.

pnpm run bench (test/bench/search.bench.ts): ASCII baseline, Cyrillic and CJK 2.3×, emoji 2.0×, an all-flag run 6.1×.

The snap was the easy half

Every defect review found was in the bookkeeping around it, not in the segmentation — and each one reached the user as duplicated or vanished text rather than as an error, because substring() swaps a range it finds reversed and clamps one it finds out of range instead of throwing.

Four, in the order they were found:

defect symptom found in
a region the clamps left empty still emitted its tags bare <mark></mark>, highlight lost round 1
a region past the end of the value bypassed that guard — the comparison did not clamp where substring() did same empty mark, frozen cursor swallowing every later region, and a trailing mark that drove truncation into eating the text before it round 2
a region nested inside an earlier one ends behind the cursor [[0,5],[1,2]] on abcdefgh gave <mark>abcdef</mark>defdefgh — the overlap emitted three times round 3
a non-integer bound NaN compares false against every guard including end > start, then lands in the cursor, where substring(NaN) reads as substring(0) and repeats the whole value round 3

All four are guarded, each by a test verified to fail with its guard removed. Both round-3 defects are reachable through CommandPaletteGroup.postFilter, which lets a caller supply its own matches — Fuse itself always emits sorted, merged, integer-bounded regions.

indices are now filtered to integer bounds and sorted before use. The sort's tie-break puts the longest of an equal-start pair first, so the outer region is marked whole instead of being split across two <mark>s.

One clamp went the other way. Mutation testing showed Math.min(…, value.length) on start was unreachable — start can only exceed the value by exceeding end, which is checked — so it was removed rather than left as an untested guard.

Coverage

Every constant and branch was verified by removing it and watching a named test fail. That is what closed the two gaps this PR previously listed as known-and-unfixed:

  • The CRLF carve-out had no fixture. It is the only cluster rule Intl.Segmenter never sees, because the < U+0300 screen answers that boundary first — so all three of its mutations passed the whole file.
  • The four surrogate range constants were pinned in one direction only. Every fixture held a real pair, which catches a range being narrowed and not one being widened; widening needs two units that are not a pair, i.e. malformed UTF-16, which reaches highlight() through any label built from a byte-truncated field. All four widening mutations passed.

One mutation still survives by design: removing the < U+0300 screen entirely. It is a performance guard and changes no output — the bench is what measures it.

Remaining caveat

The bench is advisory: no assertions, not in CI.

Notes

truncateHTMLFromStart counts retained code units and slices once, because the snap needs an index to move. Verified equivalent over 400,078 fuzzed inputs, including strings beginning with ....

A fixture trap is recorded in the tests: UAX #29 glues a run of bare emoji modifiers into a single cluster, so '\u{1F3FF}'.repeat(20) is one character, not twenty.

Checklist

  • I have linked an issue or discussion.
  • I have updated the documentation accordingly.

Full suite green on the rebase — 6304 passed, 6 skipped, 270 files. Each new case was verified failing against the unpatched version first. eslint ., vue-tsc --noEmit and pnpm build clean.

…code points

Code points are not what a reader sees. A flag is two regional indicators, a
family emoji several joined by ZWJ, `कि` a consonant plus a vowel sign. Cutting
inside one produces a *different* character rather than a broken one — 🇺🇸
truncated by a single code point re-pairs into 🇸🇺, a different country, with
nothing to signal the loss.

Both boundaries snap off the straddled cluster: the mark insertion in
`generateHighlightedText` and the truncation in `truncateHTMLFromStart`.

Chosen on measurement. Segmenting the whole string costs 455 µs at 979
characters and 52 ms at 100k. A fixed ±64 window is constant-time but wrong: a
run of flags is longer than the window, so segmentation starts mid-run and
re-pairs the indicators — reproducing the exact bug, 28 disagreements in 1044
probes. `Segments.containing()` is correct, so it ships, behind three guards: a
`< U+0300` screen, which keeps ASCII and Latin-1 off the segmenter but does
nothing for Cyrillic or CJK, both above the floor; one segmenter view per value,
built lazily; and an 8192-character ceiling, past which only the surrogate snap
applies and every multi-code-point cluster loses protection.

Two defects found in review are folded in. A region left empty by the ordering
clamps emitted a bare `<mark></mark>` with the highlight lost. A region past the
end of the value bypassed that guard entirely, because `substring()` clamps its
own arguments while the comparison did not — it emitted the same empty mark,
froze the cursor past every later region, and the trailing mark then drove
truncation into eating the text before it.

`indices` are sorted before use. That is a no-op for the search path, since Fuse
merges and sorts them itself, but `highlight()` is a published export and
`CommandPaletteGroup.postFilter` lets a caller supply its own matches;
out-of-order boundaries make `containing()` re-walk a long run of regional
indicators from its start, costing as much as the per-boundary segmenter this
avoids.

`truncateHTMLFromStart` counts retained code units and slices once, because the
snap needs an index to move. Verified equivalent over 400,078 fuzzed inputs,
including strings that themselves begin with `...`.

One fixture trap is recorded in the tests: UAX #29 glues a run of bare emoji
modifiers into a single cluster, so `'\u{1F3FF}'.repeat(20)` is one character
rather than twenty.

Closes #364

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
@IgorShevchik
IgorShevchik force-pushed the claude/search-grapheme-clusters branch from b284e71 to db194cf Compare August 13, 2026 09:18
@IgorShevchik
IgorShevchik merged commit 54b93e3 into main Aug 13, 2026
1 check passed
@IgorShevchik
IgorShevchik deleted the claude/search-grapheme-clusters branch August 13, 2026 09:40
IgorShevchik pushed a commit that referenced this pull request Aug 13, 2026
`GRAPHEME_SNAP_MAX_LENGTH` is documented, and was tested, as a bound on the
field text. It was not measured against it: `truncateHTMLFromStart` built its
snapper from `content` — the value after escaping and after `<mark>` insertion —
so the ceiling was reached early, by an amount nothing in the constant hints at.
Escaping has no fixed overhead: `&` expands to five characters, `"` to six.

    'a'.repeat(8140) + FLAG.repeat(10) + 'match'     // value.length 8185
    '&'.repeat(1600) + FLAG.repeat(50) + 'match'     // value.length 1805

Both degraded, and both rendered `🇸🇺` with an orphaned regional indicator where
the author wrote `🇺🇸` — a different country, silently. That is the defect #364
describes and #371 exists to prevent, reappearing above a threshold no reader
could predict.

`createClusterSnapper` now takes the length to weigh separately from the string
to segment. The parameter is required rather than defaulted: a default is what
would let the next derived-string caller inherit this bug without noticing.

The cost is real and is worth naming. The snapper builds a fresh `Segments` view
per call and the cut sits near the end of `html`, where `containing()` is
dearest: ~140µs at the ceiling, ~930µs on the six-fold expansion an escape-heavy
value produces there. Both are paid only when the boundary characters clear
`CLUSTER_CONTINUATION_FLOOR`, so plain text returns before a view is built and
escaped runs like `&quot;` are themselves ASCII — the dear case needs a long
value *and* a cluster landing on the cut. A fraction of a millisecond, once per
truncated field, against rendering a flag as the wrong country.

The two ceiling tests moved off `value.length + MARKUP` onto `value.length`,
which makes `snaps clusters exactly at the ceiling` the plain-text repro — a
value of exactly the advertised length previously degraded. The new case covers
the unbounded half: every fixture in `degraded paths` was free of escapable
characters, so the expansion never reached the ceiling arithmetic.

`.sync/PORTING.md`'s guard entry said the threshold "is crossed 13 characters
earlier than a reading of `value.length` suggests". That described the defect,
not the design, and is corrected here rather than left for the documentation
follow-up — a porter reading it could otherwise restore the bug on purpose.

Closes #387.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik pushed a commit that referenced this pull request Aug 13, 2026
`GRAPHEME_SNAP_MAX_LENGTH` is documented, and was tested, as a bound on the
field text. It was not measured against it: `truncateHTMLFromStart` built its
snapper from `content` — the value after escaping and after `<mark>` insertion —
so the ceiling was reached early, by an amount nothing in the constant hints at.
Escaping has no fixed overhead: `&` expands to five characters, `"` to six.

    'a'.repeat(8140) + FLAG.repeat(10) + 'match'     // value.length 8185
    '&'.repeat(1600) + FLAG.repeat(50) + 'match'     // value.length 1805

Both degraded, and both rendered `🇸🇺` with an orphaned regional indicator where
the author wrote `🇺🇸` — a different country, silently. That is the defect #364
describes and #371 exists to prevent, reappearing above a threshold no reader
could predict.

`createClusterSnapper` now takes the length to weigh separately from the string
to segment, named `fieldTextLength` for what it is. The parameter is required
rather than defaulted: a default is what would let the next derived-string
caller inherit this bug without noticing.

Segmenting a string the ceiling would have excluded is affordable, but not
because the call is cheap in isolation — on a long run of regional indicators it
is not. It is that `Array.from(html)` already paid the larger price on the same
string, unconditionally, on both sides of this fix. End to end the difference is
tens of microseconds either way, within noise. `test/bench/search.bench.ts`
gains a case that reaches the ceiling by escaping rather than by length, so the
claim has a durable measurement behind it rather than a throwaway script.

The two ceiling tests moved off `value.length + MARKUP` onto `value.length`,
which makes `snaps clusters exactly at the ceiling` the plain-text repro — a
value of exactly the advertised length previously degraded. The new case covers
the unbounded half: every fixture in `degraded paths` was free of escapable
characters, so the expansion never reached the ceiling arithmetic.

`.sync/PORTING.md`'s guard entry said the threshold "is crossed 13 characters
earlier than a reading of `value.length` suggests". That described the defect,
not the design, and is corrected here rather than left for the documentation
follow-up — a porter reading it could otherwise restore the bug on purpose.

Closes #387.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik added a commit that referenced this pull request Aug 13, 2026
#388)

`GRAPHEME_SNAP_MAX_LENGTH` is documented, and was tested, as a bound on the
field text. It was not measured against it: `truncateHTMLFromStart` built its
snapper from `content` — the value after escaping and after mark insertion — so
the ceiling was reached early, by an amount nothing in the constant hints at.
Escaping has no fixed overhead: `&` expands to five characters, `"` to six.

A value of 8185 characters, and one of 1805 made largely of `&`, both degraded
and rendered a flag as a different country, silently. That is the defect #364
describes and #371 exists to prevent, reappearing above a threshold no reader
could predict.

`createClusterSnapper` now takes the length to weigh separately from the string
to segment, named `fieldTextLength`. The parameter is required rather than
defaulted: a default is what would let the next derived-string caller inherit
this bug without noticing.

Segmenting a string the ceiling would have excluded is affordable, but not
because the call is cheap in isolation. It is that `Array.from(html)` already
paid the larger price on the same string, unconditionally, on both sides of this
fix. End to end the difference is tens of microseconds either way, within noise.
`test/bench/search.bench.ts` gains a case that reaches the ceiling by escaping
rather than by length.

`.sync/PORTING.md`'s guard entry said the threshold "is crossed 13 characters
earlier than a reading of `value.length` suggests". That described the defect,
not the design, and is corrected here — a porter reading it could otherwise
restore the bug on purpose.

Closes #387.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik pushed a commit that referenced this pull request Aug 15, 2026
…nput

`sanitizeSnippet` preserved real `<mark>` tags by swapping them for
`\0markO\0`/`\0markC\0`, escaping everything, then swapping back. NUL is an
ordinary character a snippet can carry, so what decided whether markup was
emitted was a string the input could supply.

A whole sentinel forged a tag outright:

    sanitizeSnippet('before \0markO\0 after')
    // → 'before <mark> after'      — one <mark> out, none in

    sanitizeSnippet('\0markC\0\0markO\0')
    // → '</mark><mark>'            — a close ahead of its opener

Worse, and easier: **six** of the seven bytes, immediately before a *real* tag,
were enough. The placeholder this function inserted for that tag completed the
prefix, and the restore step then found a sentinel spanning the two:

    sanitizeSnippet('a\0markO<mark>hit</mark>b')
    // → 'a<mark>markO\0hit</mark>b'

The genuine highlight moved, and with text on both sides it landed on text it
was never meant to mark. No crafted sequence — one stray fragment ahead of any
highlight. The first description of this defect, in #391 and in this branch's
first revision, said the input had to carry the sentinel itself. That understated
both the trigger and the consequence; an independent fuzzing pass found the
prefix case.

The function's own doc says the preserved tag is hardcoded *"on purpose: taking
it as a parameter would let a caller pass any tag through to the `v-html` that
renders the result."* The intent was right; the mechanism did not hold it.

Not XSS, and the bound is worth stating because it is what keeps this a content
bug: `escapeHTML` ran before the swap back, so content inside a forged region
stayed escaped, and the emitted tag came from a fixed literal rather than from a
capture group — no attribute could land inside it. Forgeable surface: the two
strings, nothing else. The consequence is spoofed emphasis, relocated highlights
and unbalanced markup reaching `v-html`, in snippets that come from whatever
backend the host application passes to `<ContentSearch>`
(`useContentSearch.ts:141,144`).

Splitting on the real tags removes the guess, and with it the collision. Both
implementations were run over ~2.1M generated inputs by an independent pass:
every divergence traces to this family, and the new one never emitted a tag
other than the two literals nor leaked an unescaped character. The regex has no
quantifiers, and growth is linear to 61MB.

Five tests added, each mutation-checked: dropping the regex capture, dropping the
closing-tag half of the condition, and escaping the tags themselves are all
caught; restoring the old implementation fails exactly the four forgery cases.
One pins the *bound* rather than the bug — a forged tag could never carry an
attribute — so a future rewrite cannot widen that silently.

Pre-existing since `557a5178`, the original port, and untouched by #365, #371
and #388. The function came from upstream unchanged, so `nuxt/ui` very likely
carries it too.

Closes #391.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik pushed a commit that referenced this pull request Aug 15, 2026
…nput

`sanitizeSnippet` preserved real `<mark>` tags by swapping them for
`\0markO\0`/`\0markC\0`, escaping everything, then swapping back. NUL is an
ordinary character a snippet can carry, so what decided whether markup was
emitted was a string the input could supply.

A whole sentinel forged a tag outright:

    sanitizeSnippet('before \0markO\0 after')
    // → 'before <mark> after'      — one <mark> out, none in

    sanitizeSnippet('\0markC\0\0markO\0')
    // → '</mark><mark>'            — a close ahead of its opener

Worse, and easier: **six** of the seven bytes, immediately before a *real* tag,
were enough. The placeholder this function inserted for that tag completed the
prefix, and the restore step then found a sentinel spanning the two:

    sanitizeSnippet('a\0markO<mark>hit</mark>b')
    // → 'a<mark>markO\0hit</mark>b'

The genuine highlight moved, and with text on both sides it landed on text it
was never meant to mark. No crafted sequence — one stray fragment ahead of any
highlight. The first description of this defect, in #391 and in this branch's
first revision, said the input had to carry the sentinel itself. That understated
both the trigger and the consequence; an independent fuzzing pass found the
prefix case.

The function's own doc says the preserved tag is hardcoded *"on purpose: taking
it as a parameter would let a caller pass any tag through to the `v-html` that
renders the result."* The intent was right; the mechanism did not hold it.

Not XSS, and the bound is worth stating because it is what keeps this a content
bug: `escapeHTML` ran before the swap back, so content inside a forged region
stayed escaped, and the emitted tag came from a fixed literal rather than from a
capture group — no attribute could land inside it. Forgeable surface: the two
strings, nothing else. The consequence is spoofed emphasis, relocated highlights
and unbalanced markup reaching `v-html`, in snippets that come from whatever
backend the host application passes to `<ContentSearch>`
(`useContentSearch.ts:141,144`).

Splitting on the real tags removes the guess, and with it the collision. Both
implementations were run over ~2.1M generated inputs by an independent pass:
every divergence traces to this family, and the new one never emitted a tag
other than the two literals nor leaked an unescaped character. The regex has no
quantifiers, and growth is linear to 61MB.

Five tests added, each mutation-checked: dropping the regex capture, dropping the
closing-tag half of the condition, and escaping the tags themselves are all
caught; restoring the old implementation fails exactly the four forgery cases.
One pins the *bound* rather than the bug — a forged tag could never carry an
attribute — so a future rewrite cannot widen that silently.

Pre-existing since `557a5178`, the original port, and untouched by #365, #371
and #388. The function came from upstream unchanged, so `nuxt/ui` very likely
carries it too.

Closes #391.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik added a commit that referenced this pull request Aug 15, 2026
…nput (#405)

`sanitizeSnippet` preserved real `<mark>` tags by swapping them for
`\0markO\0`/`\0markC\0`, escaping, then swapping back. NUL is an ordinary
character a snippet can carry, so what decided whether markup was emitted was a
string the input could supply.

A whole sentinel forged a tag outright. Worse and easier: six of its seven bytes
immediately before a real tag were enough, because the placeholder inserted for
that tag completed the prefix — so `a\0markO<mark>hit</mark>b` came back as
`a<mark>markO\0hit</mark>b`, the genuine highlight landing on text it was never
meant to mark. No crafted sequence, one stray fragment ahead of any highlight.

Not XSS: escaping ran before the swap back, so content inside a forged region
stayed escaped, and the emitted tag came from a fixed literal rather than a
capture group — no attribute could land inside it. Forgeable surface was the two
strings and nothing else. The consequence is spoofed emphasis, relocated
highlights and unbalanced markup reaching `v-html`, in snippets that come from
whatever backend the host application passes to `<ContentSearch>`.

Splitting on the real tags removes the guess, and with it the collision.
Independently fuzzed at ~2.1M inputs against both the old implementation and a
reference scanner using neither `split` nor `replaceAll`: no bypass, no
unescaped character, no tag other than the two literals. The regex has no
quantifiers; growth is linear to 61MB and costs 2.15µs against the old 2.09µs at
realistic snippet size.

Five tests added, each mutation-checked by running it: dropping the regex
capture, dropping either half of the condition, escaping the tags themselves and
returning the input untouched are all caught, and restoring the old
implementation fails exactly the forgery cases.

Pre-existing since `557a5178`, the original port; untouched by #365, #371 and
#388.

Closes #391.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(search): truncation splits grapheme clusters — a flag renders as a different country

2 participants