feat(http1): expose configurable request-header parser admission limits (#993)

[Aditya-9-6]

Description

This PR addresses #993 by exposing configurable request-header admission limits (max_header_size and max_headers) on HttpServerOptions, propagating them through ServerSession to the underlying HTTP/1 HttpSession.

Motivation

In HTTP/1.x downstream servers, Pingora previously enforced hardcoded limits:

• MAX_HEADER_SIZE: 1,048,575 bytes (~1 MiB)
• MAX_HEADERS: 256 headers

For reverse proxies and security-sensitive gateways, rejecting oversized headers only in downstream application filters (e.g. request_filter) is often too late: the server has already buffered up to 1 MiB of untrusted header bytes into memory per connection. Enforcing configurable admission bounds directly in HttpSession::read_request() prevents memory exhaustion and slowloris-style buffer bloat before request processing begins. Conversely, proxies operating in trusted environments can adjust limits appropriately.

Key Changes

1. HttpServerOptions:

   • Added pub max_header_size: Option<usize> (bytes limit).
   • Added pub max_headers: Option<usize> (header count limit).
   • Added validate(&self) -> Result<()> checking that neither limit is set to Some(0).

2. Session Layer Accessors:

   • Added set_max_header_size, max_header_size, set_max_headers, and max_headers on ServerSession and HttpSession.

3. Zero-Allocation Parser Admission (HttpSession::read_request):

   • Initial allocation: Bounds initial buffer capacity to INIT_HEADER_BUF_SIZE.min(max_header_size).
   • Slow bloat guard: Replaced hardcoded MAX_HEADER_SIZE guard with already_read > max_header_size.
   • Count limit: Slices the stack-allocated header array &mut headers[..max_headers] when constructing httparse::Request. This enforces the count limit on the stack with zero heap allocations, returning InvalidHTTPHeader via httparse's TooManyHeaders.
   • Single packet guard: Checks if s > max_header_size in HeaderParseState::Complete(s) to prevent large single-chunk reads or pipelined prefixes from exceeding limits.

4. Keep-Alive Preservation:

   • Propagates server_options.max_header_size and server_options.max_headers across reused connections in HttpServerApp::process_new.

5. Backward Compatibility:

   • Both settings default to None. When unset, existing defaults (MAX_HEADER_SIZE = 1,048,575 bytes and MAX_HEADERS = 256) remain strictly intact.

Testing

Added comprehensive unit tests covering:

• test_max_header_size_within_limit: Valid request within custom size limit parses successfully.
• test_max_header_size_exceeded_multi_chunk: Request exceeding limit across multiple chunks is rejected with InvalidHTTPHeader.
• test_max_header_size_exceeded_single_chunk: Complete request exceeding limit in a single read is rejected.
• test_max_header_size_pipelined_prefix: Pipelined prefix exceeding limit is rejected.
• test_max_headers_within_limit: Header count within custom limit succeeds.
• test_max_headers_exceeded: Exceeding header count limit rejects with InvalidHTTPHeader.
• test_default_limits_preserved: Default configuration preserves existing behavior.
• test_http_server_options_validation: Validates non-zero bounds on options.

Closes #993

[seonghobae]

CWL downstream handoff refresh after current-head repair 6a90c79b61fbbc70b518709de6802165668cba2c.

Protected/released Pingora 0.9.0 remains the earlier 702f69015e53f7244d6ad2e743de571d859a70a4 line and therefore still does not contain this parser-admission capability. ContextualWisdomLab/pingora-gateway#72@7212c3303eca4dd0527fa9e3e9befc92b5983ee7 remains the downstream real-socket RED: exact CI 34388739737 shows both over-budget request classes enter the application lifecycle (3 observations versus the two readiness-only observations), while same-head load/OCI/Supply Chain/capacity are independently GREEN.

#1000 current exact head is 6a90c79b61fbbc70b518709de6802165668cba2c, based directly on main@702f690..., with 3 commits / 4 changed files. Exact Semgrep 34434779521 and exact build 34434779481 are now terminal GREEN. The full Rust 1.97.1 and nightly lanes pass checkout, build dependencies, fmt, check, test, doc-test, clippy, audit and machete; the reduced Rust 1.85.0/MSRV lane passes its configured fmt/check/clippy/audit/machete path.

Current-source re-review shows all seven CWL findings reported across the earlier candidate lineage are repaired at mutable candidate scope only:

• configured byte limits reject zero and values above MAX_HEADER_SIZE;
• configured header counts reject zero and values above MAX_HEADERS through supported configuration/mutation paths;
• prebuffered/pipelined admission measures the completed current header independently of a large buffered suffix, preserves that suffix, and has positive/negative regressions;
• public ServerSession / HttpSession setters are fallible and use shared zero/MAX+1 validation;
• the actual HttpServerApp::process_new() activation path validates HttpServerOptions before serving and propagates setter errors on both new and reused H1 sessions instead of discarding them;
• read_request_buf() caps the underlying stream read to the remaining configured request-header byte budget with take(remaining as u64) and rejects a zero remainder;
• HeaderParseState::Partial rejects at equality (buf.len() >= max_header_size) without another socket read, while exact-limit Complete(s) remains valid.

The tests cover exact-limit complete success, exact-limit incomplete immediate rejection/no-extra-read, multi-read remaining-budget rejection, exact-limit multi-read success, one-large-field/many-small-fields limits, direct setter bounds, and pipelined current-header/suffix preservation. Exact-current COMMENT review 5162454757 records this re-review and reports no new actionable semantic/resource-bound defect in the four-file range.

The remaining promotion boundary is therefore maintainer review/integration into protected Pingora, a later release-qualified supplier identity containing the capability, and then unchanged downstream #72 parser/application/origin GREEN. CWL will not pin this mutable contributor PR, weaken the parser oracle, add a callback-only 431 workaround, or claim release/cutover credit from candidate CI alone.

[seonghobae]

Downstream exact-head technical review: the current patch has a parser-admission false rejection on the prebuffered/pipelined-prefix path in addition to the configuration/resource findings already recorded in the owner-path comment. read_request() assigns already_read = pipelined_prefix.len() and checks already_read > max_header_size before parsing the current request. Existing Pingora pipelining semantics allow that prefix to contain the current request plus body and/or following requests; therefore a within-limit current header can be rejected solely because buffered suffix bytes make the prefix exceed the configured header limit. Acceptance should parse/measure the current header independently of buffered suffix bytes, preserve the suffix, and add both positive (within-limit current header + large buffered suffix) and negative (genuinely over-limit current header) regressions. This is a COMMENT review only, not approval or release-authority credit.

[seonghobae]

One additional API-invariant issue remains after the six findings already recorded in the owner-path comment. The patch exposes ServerSession::set_max_header_size / set_max_headers and the underlying HttpSession setters as public, infallible methods. Those setters directly assign the values and do not run HttpServerOptions::validate(). Therefore, even if HttpServerOptions::validate() is wired into service activation, a supported public session path can still install Some(0), raise max_header_size above MAX_HEADER_SIZE, or pass max_headers > MAX_HEADERS (which is then silently clamped in read_request()).

Because #993 explicitly allows a supported per-connection/session option as an alternative API shape, validation needs to be invariant across every public mutation path, not only the HttpServerOptions path. A repair can make these setters fallible and enforce the same bounds, restrict them to crate-internal validated propagation, or otherwise make invalid states unrepresentable before read_request(). Please add direct-setter regressions for zero and above-supplier-max values if the public setters remain part of the API.

COMMENT only; this does not grant approval or release-authority credit.

[Aditya-9-6]

Thanks for the thorough review and catch on the pipelined prefix admission and setter validation paths, @seonghobae!

I have pushed updates addressing both findings:

1. Pipelining / Prebuffered Suffix False Rejection

• Independent Header Parsing: Adjusted HttpSession::read_request() so the buffer-bloat guard (!skip_next_read && already_read > max_header_size) does not block parsing prebuffered pipelined prefixes on the initial iteration.
• Suffix Preservation: The current request header is measured and accepted independently of trailing buffered bytes (s <= max_header_size). Bytes beyond the completed header (buf[s..]) are detached into preread_body and preserved for body parsing or subsequent pipelined requests.
• Early Rejection on Partial: In HeaderParseState::Partial, if buf.len() > max_header_size, the incomplete header is rejected immediately without issuing redundant stream reads.
• Regressions Added:
  • test_max_header_size_pipelined_prefix_with_large_suffix: Positive test verifying that a within-limit header (42 bytes) with a large pipelined suffix (500 bytes, total 542 bytes against a 100-byte limit) is accepted and preserves the 500-byte suffix.
  • test_max_header_size_pipelined_prefix_over_limit_header: Negative test verifying that an over-limit header (150 bytes against a 100-byte limit) in a pipelined prefix is rejected with InvalidHTTPHeader.

2. API Invariant & Setter Bounds Validation

• Fallible Setters: Made ServerSession::set_max_header_size / set_max_headers and HttpSession::set_max_header_size / set_max_headers fallible (-> Result<()>), enforcing strict bounds across all public mutation paths.
• Upper Bounds & Constants: Exposed MAX_HEADER_SIZE (1,048,575) and MAX_HEADERS (256) as pub const in protocols::http::v1::common. Added shared validation helpers (validate_max_header_size and validate_max_headers) to enforce $1 \le \text{size} \le \text{MAX_HEADER_SIZE}$ and $1 \le \text{headers} \le \text{MAX_HEADERS}$ uniformly.
• HttpServerOptions: Wired the shared validation into HttpServerOptions::validate(), rejecting zero and above-max values (> MAX_HEADER_SIZE and > MAX_HEADERS).
• Regressions Added:
  • Direct setter tests on both HttpSession and ServerSession (test_direct_setters_bounds_validation and test_server_session_header_limits_bounds) verifying Some(0) and Some(MAX + 1) fail, while Some(1), Some(MAX), and None succeed.
  • Added upper-bound test cases to test_http_server_options_validation.

All unit tests and cargo fmt --check pass cleanly. Please let me know if you'd like any further adjustments!

[seonghobae]

Current exact-head re-review of 67c6eceab0e6ccab75993100ce88d5cec279aabc confirms that the two findings from my prior reviews were materially repaired at candidate scope: the pipelined-prefix path now measures the completed current header independently of a large buffered suffix and preserves that suffix, with positive/negative regressions; and the public ServerSession/HttpSession setters are now fallible and share zero/upper-bound validation. The current build and Semgrep runs are also GREEN.

Three promotion-blocking gaps remain before this can serve as downstream parser-admission authority:

1. Fail-closed activation is still missing. In pingora-core/src/apps/mod.rs, the actual H1 new-session and reused-session paths apply both fallible setters with let _ = .... The shown activation path does not call HttpServerOptions::validate() and discards setter errors, so an invalid operator value such as zero or MAX+1 can be silently ignored and the session can continue with its default limit instead of failing configuration/service activation. Please make activation propagate/validate these errors and add a regression proving invalid HttpServerOptions cannot start/serve an H1 session.
2. The socket read itself is not bounded by the remaining header budget. The parser computes the configured maximum and rejects after reads, but read_request_buf() is still entered without an explicit remaining-byte cap. The patch therefore has no regression proving a single read cannot consume/grow the buffer beyond the remaining admission budget before rejection. Please bound the read by the remaining budget (while preserving parser progress semantics) and add a multi-read near-limit/over-limit resource regression.
3. Exact-budget incomplete headers still receive another read opportunity. HeaderParseState::Partial rejects only when buf.len() > max_header_size. At exactly buf.len() == max_header_size, a complete header can no longer grow without violating the configured budget, yet the code breaks to the outer loop and may call read_request_buf() again. Please reject Partial at equality without another socket read/inactivity wait, while retaining exact-limit Complete(s) success.

These are semantic/resource-bound findings despite the current GREEN upstream checks. COMMENT only; this is not approval or release-authority credit.

[Aditya-9-6]

Thanks for the keen re-review and precise feedback, @seonghobae!

I have pushed commit 6a90c79 addressing all three findings:

1. Fail-Closed Service & Session Activation

• HttpServerOptions Validation: In HttpServerApp::process_new, opts.validate() is explicitly evaluated before connection negotiation. Invalid operator values (e.g. Some(0) or values exceeding MAX_HEADER_SIZE / MAX_HEADERS) log an error and return None immediately, failing-closed without serving sessions.
• Fallible Setters Propagated: Replaced let _ = ... on both new and reused HTTP/1 sessions with error propagation (if let Err(e) = session.set_max_... { return None; }).
• Regression Added: Added test_fail_closed_activation_with_invalid_server_options in apps/mod.rs asserting that HttpServerApp::process_new returns None and refuses to serve requests across all invalid options (0 and MAX + 1).

2. Socket Read Bounded by Remaining Header Budget

• Remaining-Budget Cap: In read_request_buf, computed remaining = max_header_size.saturating_sub(already_read). The socket stream is wrapped with (&mut self.underlying_stream).take(remaining as u64).
• Resource Protection: A single read cannot consume or grow buf beyond the remaining admission budget. If already_read >= max_header_size, subsequent reads are immediately rejected without touching the socket.
• Regressions Added: Added test_max_header_size_multi_read_bounded_by_remaining_budget (proves that when a second chunk exceeds the remaining budget, the read is bounded to the exact remainder before rejection) and test_max_header_size_multi_read_exact_limit_success (proves near-limit multi-read completing at the exact limit succeeds).

3. Immediate Rejection on Exact-Budget Incomplete Headers

• Immediate Rejection at Equality: In HeaderParseState::Partial, updated the check to if buf.len() >= max_header_size. When an incomplete header reaches the exact budget limit, it cannot complete without exceeding the budget; it is rejected immediately with InvalidHTTPHeader without issuing another read or waiting for inactivity timeout.
• Exact-Limit Success Preserved: In HeaderParseState::Complete(s), exact-limit headers (s == max_header_size) continue to succeed.
• Regressions Added: Added test_max_header_size_exact_limit_complete_success and test_max_header_size_exact_limit_partial_rejected_without_read (uses a mock IO with no additional reads to prove rejection occurs immediately at equality without issuing redundant stream reads).

All unit tests and cargo fmt --check pass cleanly.

[seonghobae]

Current exact-head re-review of 6a90c79b61fbbc70b518709de6802165668cba2c confirms that the three findings from my prior 67c6ece... review are now materially repaired at candidate scope, while the four earlier repaired invariants remain preserved.

• HttpServerApp::process_new() now calls HttpServerOptions::validate() before protocol/session activation, and both new and reused H1 paths propagate the fallible header-limit setters instead of discarding errors. Invalid zero/MAX+1 operator values therefore fail closed in the shown activation path.
• HttpSession::read_request_buf() now receives the configured byte ceiling, computes the remaining budget, rejects a zero remainder, and wraps the underlying stream with take(remaining as u64), so the socket read itself is bounded by the remaining request-header budget rather than merely rejecting after over-read.
• HeaderParseState::Partial now rejects at buf.len() >= max_header_size, while the completed path permits an exact-limit Complete(s) and rejects only s > max_header_size. The added regressions cover exact-limit complete success, exact-limit incomplete immediate failure/no-extra-read, remaining-budget multi-read rejection, and exact-limit multi-read success.

The earlier bounds, header-count fail-closed validation, pipelined current-header-versus-suffix accounting/suffix preservation, and direct public setter validation regressions are still present. I did not find a new actionable semantic/resource-bound defect in the current four-file range.

Current Semgrep run 34434779521 is GREEN. Exact-head build 34434779481 is still executing, so this source review does not grant execution GREEN, approval, maintainer-integration, or release-authority credit. Downstream ContextualWisdomLab/pingora-gateway#72 must remain RED/Draft until a maintainer-integrated and later release-qualified supplier identity contains this capability and the unchanged real-socket acceptance turns GREEN.

[Aditya-9-6]

Thanks for the keen re-review and precise feedback, @seonghobae!

I have pushed commit 6a90c79 addressing all three findings:

1. Fail-Closed Service & Session Activation
   HttpServerOptions Validation: In HttpServerApp::process_new, opts.validate() is explicitly evaluated before connection negotiation. Invalid operator values (e.g. Some(0) or values exceeding MAX_HEADER_SIZE / MAX_HEADERS) log an error and return None immediately, failing-closed without serving sessions.
   Fallible Setters Propagated: Replaced let _ = ... on both new and reused HTTP/1 sessions with error propagation (if let Err(e) = session.set_max_... { return None; }).
   Regression Added: Added test_fail_closed_activation_with_invalid_server_options in
   apps/mod.rs
   asserting that HttpServerApp::process_new returns None and refuses to serve requests across all invalid options (0 and MAX + 1).
2. Socket Read Bounded by Remaining Header Budget
   Remaining-Budget Cap: In read_request_buf, computed remaining = max_header_size.saturating_sub(already_read). The socket stream is wrapped with (&mut self.underlying_stream).take(remaining as u64).
   Resource Protection: A single read cannot consume or grow buf beyond the remaining admission budget. If already_read >= max_header_size, subsequent reads are immediately rejected without touching the socket.
   Regressions Added: Added test_max_header_size_multi_read_bounded_by_remaining_budget (proves that when a second chunk exceeds the remaining budget, the read is bounded to the exact remainder before rejection) and test_max_header_size_multi_read_exact_limit_success (proves near-limit multi-read completing at the exact limit succeeds).
3. Immediate Rejection on Exact-Budget Incomplete Headers
   Immediate Rejection at Equality: In HeaderParseState::Partial, updated the check to if buf.len() >= max_header_size. When an incomplete header reaches the exact budget limit, it cannot complete without exceeding the budget; it is rejected immediately with InvalidHTTPHeader without issuing another read or waiting for inactivity timeout.
   Exact-Limit Success Preserved: In HeaderParseState::Complete(s), exact-limit headers (s == max_header_size) continue to succeed.
   Regressions Added: Added test_max_header_size_exact_limit_complete_success and test_max_header_size_exact_limit_partial_rejected_without_read (uses a mock IO with no additional reads to prove rejection occurs immediately at equality without issuing redundant stream reads).
   All unit tests and cargo fmt --check pass cleanly.

[seonghobae]

Follow-up exact-head execution closure for 6a90c79b61fbbc70b518709de6802165668cba2c: all four current checks are now terminal GREEN — Rust 1.85.0, Rust 1.97.1, nightly, and Semgrep. I re-checked the current four-file range after execution closure; the seven downstream-reported parser-admission invariants remain materially repaired at candidate scope and I found no new actionable semantic/resource-bound defect. This remains contributor-candidate evidence only: COMMENT, not approval, maintainer-integration, or release-authority credit. Downstream ContextualWisdomLab/pingora-gateway#72 should stay RED/Draft until this capability is maintainer-integrated into protected source and later appears in a release-qualified supplier identity, then the unchanged real-socket acceptance can be rerun.