Skip to content

fix(CommandPalette): keep astral characters intact when truncating search results - #365

Merged
IgorShevchik merged 3 commits into
mainfrom
claude/search-astral-truncation
Aug 11, 2026
Merged

fix(CommandPalette): keep astral characters intact when truncating search results#365
IgorShevchik merged 3 commits into
mainfrom
claude/search-astral-truncation

Conversation

@IgorShevchik

@IgorShevchik IgorShevchik commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Linked issue

Closes #339. Supersedes #347.

Type of change

  • Documentation (updates to the documentation or readme)
  • Bug fix (a non-breaking change that fixes an issue)
  • Enhancement (improving an existing functionality)
  • New feature (a non-breaking change that adds functionality)
  • Chore (updates to the build process or auxiliary tools and libraries)
  • Breaking change (fix or feature that would cause existing functionality to change)

Description

This PR was rewritten after review. An earlier revision carried a second commit that replaced @arsalan507's implementation with in-place surrogate coalescing, justified by performance numbers. Five independent reviews found those numbers were measured on the helper alone, excluding an O(n) scan the same commit added to the caller; that the replacement was slower than the original on astral-dense input with an early match; and that at the project's default resultLimit: 12 the entire difference is single-digit microseconds. That commit has been dropped. What remains is @arsalan507's fix as written, plus test hardening. The earlier claims are retracted below.

Commit 1 — @arsalan507's fix, carried unchanged (12c8e741, original authorship and message preserved; content-identical to f08572d0 on #347).

truncateHTMLFromStart walked the string one UTF-16 code unit at a time. An astral character — emoji, most CJK extension blocks — occupies two, so a truncation boundary landing between them sliced the pair in half and emitted an unpaired surrogate rendering as . Deterministic from 7 characters of prefix onward, and every length after.

The fix iterates code points and measures the caller's budget in the same units, so the two halves cannot drift apart. That is exactly what #339 prescribed, clause for clause — "Iterate by code point rather than code unit … and count length the same way."

Commit 2 — test hardening only. src/ is untouched by this commit; git diff 12c8e741..HEAD --name-only is one file.

Three gaps found by running the suite against deliberately wrong implementations:

  • The fixture used one astral character, U+1F600, which sits comfortably inside both surrogate ranges, so any implementation with an off-by-one range bound passed. The sweep now covers U+10000 and U+10FFFF (first and last astral code points), U+20000 (low surrogate U+DC00, the lower edge), U+1F3FF (low surrogate U+DFFF, the upper edge, and a skin-tone modifier that appears in ordinary UI text) and U+1F600.
  • The sweep asserted only the absence of a lone surrogate — which an implementation that deletes every astral character satisfies trivially, while carrying a name saying it does not. It now asserts how many characters survived, which also closes the ?? '' coalesce that would have let a highlight() returning undefined throughout pass vacuously.
  • No case placed astral content at or after the mark, so the caller's half of the fix was uncovered: reverting it alone left every test green. A new case covers it, using the indices real fuse.js returns for the search term.

expect(result).not.toContain('�') is dropped — it could never fail, since the bug emits raw unpaired surrogates and U+FFFD only appears after a lossy re-encode that does not happen on this path. Containment checks are replaced by exact-output assertions, and the retained-prefix length is derived rather than hardcoded: it is invariantly '<mark>'.length + '</mark>'.length, because maxLength counts the tag characters the inner counter skips.

Verified against four wrong implementations — skip truncation when astral content is present, strip astral characters first, revert the caller's half, revert to code-unit iteration — each of which now fails, and all but one of which passed the previous cases.

Upstream. @arsalan507 submitted this to nuxt/ui first (nuxt/ui#6817) so src/runtime/utils/search.ts stays in sync, which was the right call — and dropping our replacement implementation preserves that property. Note separately that the file is not byte-identical with upstream today: highlight() carries a b24ui-only fifth argument useTokenSearch from c502157b / 6743f793, shipped in v2.8.0. That predates this PR by three months and was invisible to anyone not running git log --follow across the fuse.tssearch.ts rename; it is recorded as a .sync/PORTING.md invariant in #366, and its missing coverage is #363.

Scope. #339 is scoped entirely to truncateHTMLFromStart — its title, its Problem section, and its prescribed fix. This closes it. #362 (<mark> inserted inside an astral character, because Fuse's indices are code-unit offsets too) is a separate defect in a different function, not a remainder of #339. #364 (grapheme clusters) is likewise separate.

Behaviour change worth a changelog line. Not purely a bug fix: an astral character now costs one unit of the truncation budget instead of two, so 20 emoji before a 5-character match keep 13 where main kept 6 and half of a 7th. Some outputs that were already well-formed simply get a longer prefix. BMP-only content is byte-identical to main.

Checklist

  • I have linked an issue or discussion.
  • I have updated the documentation accordingly.

vitest 328 passed across test/utils/ and test/components/CommandPalette.spec.ts; eslint and vue-tsc --noEmit clean.

…arch results

`truncateHTMLFromStart` walked the highlighted snippet one UTF-16 code unit at
a time. An astral character (emoji, most CJK extension blocks) occupies two
code units, so a truncation boundary landing between them sliced the pair in
half and emitted an unpaired surrogate, rendering as `<?>`.

Iterate by code point instead, and measure the length budget the same way so
both sides stay in the same units. Behaviour for BMP-only content is unchanged.

Closes #339
…e range bounds

Hardens the cases added by the previous commit. No implementation change —
`src/runtime/utils/search.ts` is untouched.

Three gaps, found by re-running the suite against deliberately wrong
implementations:

The fixture used one astral character, U+1F600, which sits comfortably inside
both surrogate ranges — so any implementation whose range bounds are off by one
passed. The sweep now runs over U+10000 and U+10FFFF (the first and last astral
code points), U+20000 (low surrogate U+DC00, the lower edge), U+1F3FF (low
surrogate U+DFFF, the upper edge, and a skin-tone modifier that appears in
ordinary UI text) and U+1F600.

The sweep asserted only the absence of a lone surrogate, which an implementation
that deletes every astral character satisfies trivially — while carrying a name
that says it does not. It now asserts how many characters survived, which also
closes the `?? ''` coalesce that would have let a `highlight()` returning
`undefined` throughout pass vacuously.

`expect(result).not.toContain('�')` could never fail: the bug emits raw
unpaired surrogates, and U+FFFD only appears after a lossy re-encode that does
not happen on this path. Dropped.

Containment checks are replaced by exact-output assertions, and a new case
places astral content after the match so the caller's half of the fix — the
budget measured in code points — is covered; reverting that half alone
previously left every test green. Its fixture uses the indices real fuse.js
returns for the search term, `[[40, 44]]`.

The retained-prefix length is now derived rather than hardcoded. It is
invariantly `'<mark>'.length + '</mark>'.length`: `maxLength` counts the tag
characters that the counter inside `truncateHTMLFromStart` skips, so the two
cancel regardless of the match, the filler, or whether the content is BMP or
astral.

Verified against four wrong implementations — skip truncation when astral
content is present, strip astral characters first, revert the caller's half, and
revert to code-unit iteration — each of which now fails, and all but one of
which passed the previous cases.

Refs #339, #347

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(search): truncateHTMLFromStart splits astral characters, emitting lone surrogates

3 participants