Skip to content

fix: escape non utf-8 bytes in string output - #74

Merged
swordqiu merged 1 commit into
yunionio:masterfrom
swordqiu:hotfix/qj-quote-invalid-utf8
Sep 16, 2026
Merged

swordqiu merged 1 commit into
yunionio:masterfrom
swordqiu:hotfix/qj-quote-invalid-utf8

Conversation

@swordqiu

Copy link
Copy Markdown
Member

背景

quoteString 把 0x7f 以上的字节一律原样写出。值本身不是合法 UTF-8 时(来自 \x 转义、原始输入中的非法字节,或 Go 结构体里的非 UTF-8 字符串),产出的文档也就不是合法 UTF-8。

变更

按 rune 遍历字符串:

  • ASCII 仍走原有的 escapeJsonChar
  • 合法多字节序列整体写出,不做改动
  • 构不成合法序列的字节改用 \xXX 转义写出

parseQuoteString 本就支持 \x 并原样还原字节,因此值的字节级往返保持不变,既有的 TestJSONSpecialChar 无需修改。合法 UTF-8 的值不会引入 \x,输出仍是标准 JSON。

取舍

含非法字节的值,其输出需要 \x 扩展才能解析 —— 本库可解析,标准 JSON 解析器不行。这是保持字节往返所必须付出的代价(若改为替换为 U+FFFD,往返就会破坏)。

测试

新增 quote_test.go

  • TestQuoteStringRoundTrip:含非法字节、截断序列、CESU-8 代理的各种值,经 quoteString 后解析回来必须字节相同,且输出本身是合法 UTF-8
  • TestQuoteStringKeepsStandardUTF8:合法 UTF-8 值的输出可被 encoding/json 还原
  • TestMarshalInvalidUTF8RoundTripMarshalParseString 的值相等

go test ./... 全量通过,含既有的 TestJSONSpecialChar

quoteString wrote every byte above 0x7f as is, so a value that is not
valid utf-8 produced a document that is not valid utf-8 either.

Walk the string rune by rune and write the bytes that are not part of a
valid sequence as \xXX escapes, which parseQuoteString reads back
unchanged, so the value still survives a round trip.
@swordqiu
swordqiu merged commit 4b02421 into yunionio:master Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant