Skip to content

fix: decode UTF-8 in HTTP/2 header values as HTTP/1.1 does - #1500

Draft
pjfanning wants to merge 2 commits into
apache:mainfrom
pjfanning:http2-header-value-utf8
Draft

pjfanning wants to merge 2 commits into
apache:mainfrom
pjfanning:http2-header-value-utf8

Conversation

@pjfanning

Copy link
Copy Markdown
Member

Motivation:
HPACK decodes string literals as ISO-8859-1, one char per octet.
RequestParsing.parseHeaderPair then encoded the header line as UTF-8
before handing it to the HTTP/1.1 header parser, which encoded every
octet above 0x7F as a char of its own. A UTF-8 value such as "café"
came out as "café" over HTTP/2, while HTTP/1.1 gives "café".

Modification:
Encode the line as ISO-8859-1, so that the parser sees the octets that
were on the wire and applies its usual UTF-8 decoding. Octets that are
not valid UTF-8 are still copied one char per byte. This relies on the
parser keeping the line end after an invalid sequence.

Result:
Regular header values (request headers, response headers, trailers and
the joined cookie header) are decoded the same way over HTTP/2 and
HTTP/1.1.

Tests:

  • http2-tests/testOnly RequestParsingSpec client.ResponseParsingSpec:
    passed

References:
Depends on #1499; this branch contains its commit and should be rebased once that merges. Not covered: content-type, :path and :authority, which HeaderDecompression parses directly from the ISO-8859-1 string.

🤖 Generated with Claude Code

Motivation:
When a header value byte looks like the lead byte of a 2-, 3- or 4-byte
UTF-8 sequence but the bytes after it do not complete one, the header
parser copied the value as-is but still skipped the full sequence
length. At the end of a value that skipped the CR/LF, so the next header
line was read as part of this value (and a bare CR could end up being
consumed in the 2-byte case).

Modification:
On a failed decode, take only the lead byte and the continuation bytes
(10xxxxxx) directly after it as the sequence and copy them one char per
byte, as before. Any other byte, including CR and LF, is left for the
next step of the scan. The three per-length branches are folded into one.

Result:
A header value ending in a stray non-ASCII byte keeps its line end, and
the following header is parsed as its own header. Valid UTF-8 and
invalid sequences made only of continuation bytes decode as before.

Tests:
- http-core/testOnly HttpHeaderParserCRLFSpec HttpHeaderParserLFSpec:
  passed; the two new tests fail without the fix

References:
None - found while looking at HTTP/2 header value decoding
Motivation:
HPACK decodes string literals as ISO-8859-1, one char per octet.
RequestParsing.parseHeaderPair then encoded the header line as UTF-8
before handing it to the HTTP/1.1 header parser, which encoded every
octet above 0x7F as a char of its own. A UTF-8 value such as "café"
came out as "café" over HTTP/2, while HTTP/1.1 gives "café".

Modification:
Encode the line as ISO-8859-1, so that the parser sees the octets that
were on the wire and applies its usual UTF-8 decoding. Octets that are
not valid UTF-8 are still copied one char per byte. This relies on the
parser keeping the line end after an invalid sequence.

Result:
Regular header values (request headers, response headers, trailers and
the joined cookie header) are decoded the same way over HTTP/2 and
HTTP/1.1.

Tests:
- http2-tests/testOnly RequestParsingSpec client.ResponseParsingSpec:
  passed

References:
None - builds on the HTTP/1.1 header value fix
@pjfanning pjfanning added this to the 2.1.0 milestone Nov 18, 2026
@pjfanning
pjfanning marked this pull request as draft November 18, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant