fix: escape non utf-8 bytes in string output - #74
Merged
Merged
Conversation
quoteString wrote every byte above 0x7f as is, so a value that is not valid utf-8 produced a document that is not valid utf-8 either. Walk the string rune by rune and write the bytes that are not part of a valid sequence as \xXX escapes, which parseQuoteString reads back unchanged, so the value still survives a round trip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
quoteString把 0x7f 以上的字节一律原样写出。值本身不是合法 UTF-8 时(来自\x转义、原始输入中的非法字节,或 Go 结构体里的非 UTF-8 字符串),产出的文档也就不是合法 UTF-8。变更
按 rune 遍历字符串:
escapeJsonChar\xXX转义写出parseQuoteString本就支持\x并原样还原字节,因此值的字节级往返保持不变,既有的TestJSONSpecialChar无需修改。合法 UTF-8 的值不会引入\x,输出仍是标准 JSON。取舍
含非法字节的值,其输出需要
\x扩展才能解析 —— 本库可解析,标准 JSON 解析器不行。这是保持字节往返所必须付出的代价(若改为替换为 U+FFFD,往返就会破坏)。测试
新增
quote_test.go:TestQuoteStringRoundTrip:含非法字节、截断序列、CESU-8 代理的各种值,经quoteString后解析回来必须字节相同,且输出本身是合法 UTF-8TestQuoteStringKeepsStandardUTF8:合法 UTF-8 值的输出可被encoding/json还原TestMarshalInvalidUTF8RoundTrip:Marshal→ParseString的值相等go test ./...全量通过,含既有的TestJSONSpecialChar。