fix(CommandPalette): keep astral characters intact when truncating search results - #365
Merged
Merged
Conversation
…arch results `truncateHTMLFromStart` walked the highlighted snippet one UTF-16 code unit at a time. An astral character (emoji, most CJK extension blocks) occupies two code units, so a truncation boundary landing between them sliced the pair in half and emitted an unpaired surrogate, rendering as `<?>`. Iterate by code point instead, and measure the length budget the same way so both sides stay in the same units. Behaviour for BMP-only content is unchanged. Closes #339
This was referenced Aug 11, 2026
…e range bounds
Hardens the cases added by the previous commit. No implementation change —
`src/runtime/utils/search.ts` is untouched.
Three gaps, found by re-running the suite against deliberately wrong
implementations:
The fixture used one astral character, U+1F600, which sits comfortably inside
both surrogate ranges — so any implementation whose range bounds are off by one
passed. The sweep now runs over U+10000 and U+10FFFF (the first and last astral
code points), U+20000 (low surrogate U+DC00, the lower edge), U+1F3FF (low
surrogate U+DFFF, the upper edge, and a skin-tone modifier that appears in
ordinary UI text) and U+1F600.
The sweep asserted only the absence of a lone surrogate, which an implementation
that deletes every astral character satisfies trivially — while carrying a name
that says it does not. It now asserts how many characters survived, which also
closes the `?? ''` coalesce that would have let a `highlight()` returning
`undefined` throughout pass vacuously.
`expect(result).not.toContain('�')` could never fail: the bug emits raw
unpaired surrogates, and U+FFFD only appears after a lossy re-encode that does
not happen on this path. Dropped.
Containment checks are replaced by exact-output assertions, and a new case
places astral content after the match so the caller's half of the fix — the
budget measured in code points — is covered; reverting that half alone
previously left every test green. Its fixture uses the indices real fuse.js
returns for the search term, `[[40, 44]]`.
The retained-prefix length is now derived rather than hardcoded. It is
invariantly `'<mark>'.length + '</mark>'.length`: `maxLength` counts the tag
characters that the counter inside `truncateHTMLFromStart` skips, so the two
cancel regardless of the match, the filler, or whether the content is BMP or
astral.
Verified against four wrong implementations — skip truncation when astral
content is present, strip astral characters first, revert the caller's half, and
revert to code-unit iteration — each of which now fails, and all but one of
which passed the previous cases.
Refs #339, #347
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LWWrBHgfqGSbeU3V6UuMF8
IgorShevchik
force-pushed
the
claude/search-astral-truncation
branch
from
August 11, 2026 13:41
9743f3d to
c999e5d
Compare
This was referenced Aug 11, 2026
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue
Closes #339. Supersedes #347.
Type of change
Description
Commit 1 — @arsalan507's fix, carried unchanged (
12c8e741, original authorship and message preserved; content-identical tof08572d0on #347).truncateHTMLFromStartwalked the string one UTF-16 code unit at a time. An astral character — emoji, most CJK extension blocks — occupies two, so a truncation boundary landing between them sliced the pair in half and emitted an unpaired surrogate rendering as�. Deterministic from 7 characters of prefix onward, and every length after.The fix iterates code points and measures the caller's budget in the same units, so the two halves cannot drift apart. That is exactly what #339 prescribed, clause for clause — "Iterate by code point rather than code unit … and count length the same way."
Commit 2 — test hardening only.
src/is untouched by this commit;git diff 12c8e741..HEAD --name-onlyis one file.Three gaps found by running the suite against deliberately wrong implementations:
?? ''coalesce that would have let ahighlight()returningundefinedthroughout pass vacuously.fuse.jsreturns for the search term.expect(result).not.toContain('�')is dropped — it could never fail, since the bug emits raw unpaired surrogates and U+FFFD only appears after a lossy re-encode that does not happen on this path. Containment checks are replaced by exact-output assertions, and the retained-prefix length is derived rather than hardcoded: it is invariantly'<mark>'.length + '</mark>'.length, becausemaxLengthcounts the tag characters the inner counter skips.Verified against four wrong implementations — skip truncation when astral content is present, strip astral characters first, revert the caller's half, revert to code-unit iteration — each of which now fails, and all but one of which passed the previous cases.
Upstream. @arsalan507 submitted this to
nuxt/uifirst (nuxt/ui#6817) sosrc/runtime/utils/search.tsstays in sync, which was the right call — and dropping our replacement implementation preserves that property. Note separately that the file is not byte-identical with upstream today:highlight()carries a b24ui-only fifth argumentuseTokenSearchfromc502157b/6743f793, shipped in v2.8.0. That predates this PR by three months and was invisible to anyone not runninggit log --followacross thefuse.ts→search.tsrename; it is recorded as a.sync/PORTING.mdinvariant in #366, and its missing coverage is #363.Scope. #339 is scoped entirely to
truncateHTMLFromStart— its title, its Problem section, and its prescribed fix. This closes it. #362 (<mark>inserted inside an astral character, because Fuse'sindicesare code-unit offsets too) is a separate defect in a different function, not a remainder of #339. #364 (grapheme clusters) is likewise separate.Behaviour change worth a changelog line. Not purely a bug fix: an astral character now costs one unit of the truncation budget instead of two, so 20 emoji before a 5-character match keep 13 where
mainkept 6 and half of a 7th. Some outputs that were already well-formed simply get a longer prefix. BMP-only content is byte-identical tomain.Checklist
vitest328 passed acrosstest/utils/andtest/components/CommandPalette.spec.ts;eslintandvue-tsc --noEmitclean.