Skip to content

Preserve the final HTTP/2 DATA frame during graceful shutdown - #1455

Open
He-Pin wants to merge 1 commit into
apache:mainfrom
He-Pin:test/http2-graceful-credit
Open

He-Pin wants to merge 1 commit into
apache:mainfrom
He-Pin:test/http2-graceful-credit

Conversation

@He-Pin

@He-Pin He-Pin commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Motivation

Graceful HTTP/2 shutdown must deliver the final response DATA and any trailing HEADERS before completing the connection. A synchronous completion callback must not observe stale flushed state during a multiplexer transition.

Modification

Preserve PJ Fanning's production completion guard from 66ef4e5 and strengthened test from dd6f192. Document that hasFlushedAllData returns false while a transition is in progress. The depth counter guards observation only; it does not make nested state updates safe. Add normal server coverage with and without trailers and client streamed-request/trailer completion after an early final response.

Result

The final DATA and optional END_STREAM trailing HEADERS precede connection completion. This concerns stream-stage completion, not operating-system flush guarantees. The alternative notification design and nested-update questions remain open.

Tests

  • Offline Scala 2.13.18 production compilation with fatal warnings: passed.
  • Two selected server cases using the cached HTTP/2 harness: passed.
  • One selected client case using cached support classes: passed; no baseline-failure claim.
  • Cached Scalafmt 3.11.5 and git diff --check: passed.
  • The local checks were scoped, not a clean full build. GitHub Validate and test run 44180196906 completed successfully on this exact head.
    The short no-output assertion is only a smoke check; positive byte, trailer and completion assertions provide the main coverage.

References

Refs #1455.
Preserves PJ Fanning's contributions from 66ef4e5 and dd6f192.
Netty #18126 inspired ordinary lifecycle coverage; the Pekko implementation is independent.

pjfanning added a commit to He-Pin/incubator-pekko-http that referenced this pull request Nov 17, 2026
Motivation:
During graceful shutdown, if the multiplexer was already pulled
(WaitingForData) when a WINDOW_UPDATE released the last DATA frame of the
last stream, sendDataFrame pulled the frame first. Pulling closed the
stream, so completeIfDone saw hasFlushedAllData (still the old state) and
completed the stage before the frame was pushed. The push then failed
with "Cannot push closed port" and the final DATA frame was lost.

Modification:
Track multiplexer state transitions and report hasFlushedAllData as false
while one is in progress. updateState already calls onAllDataFlushed once
the new state has flushed everything.

Result:
The final DATA frame is sent before the connection completes.

Tests:
- sbt "http2-tests/testOnly ...Http2ServerSpec ...Http2ClientSpec": 195 passed, 17 pending
- The tightened graceful shutdown test in this PR fails without this change

References:
Refs apache#1455
pjfanning added a commit to He-Pin/incubator-pekko-http that referenced this pull request Nov 17, 2026
Motivation:
The test could not tell whether the server held back stream data until
the WINDOW_UPDATE arrived, and the probe's own stream window check could
not fire because it assumed the default initial window.

Modification:
Tell the probe about the 8-byte stream window, use expectDATA instead of a
manual loop, and require silence (with termination still pending) before
sending the WINDOW_UPDATE.

Result:
The test fails if the server sends past its stream credit or completes
shutdown early.

Tests:
- sbt "http2-tests/testOnly ...Http2ServerSpec ...Http2ClientSpec": 195 passed, 17 pending

References:
Refs apache#1455 review
@pjfanning
pjfanning force-pushed the test/http2-graceful-credit branch from c660cf6 to dd6f192 Compare November 17, 2026 12:46
@pjfanning

Copy link
Copy Markdown
Member

I rebased this onto main and pushed two separate commits.

  • dd6f192: addresses the review on the test. The probe is told about the 8-byte stream window, expectDATA replaces the manual loop, and the test now requires 100 ms of silence, with termination still pending, before sending the WINDOW_UPDATE.
  • 66ef4e5: the tighter test exposed a bug in Http2Multiplexer. With the 100 ms pause, the multiplexer is already pulled (WaitingForData) when the WINDOW_UPDATE arrives. sendDataFrame pulls the last DATA frame, which closes the last stream, and completeIfDone then sees hasFlushedAllData as true and completes the stage before the frame is pushed. The push fails with Cannot push closed port and the final 8 bytes are lost. The fix makes hasFlushedAllData false while a multiplexer state transition is in progress; updateState already calls onAllDataFlushed afterwards.

This PR now contains a production fix, so the title and body may need updating. Feel free to revert anything you don't agree with or split it out.

@He-Pin He-Pin changed the title Cover HTTP/2 graceful shutdown with flow-controlled response data Preserve the final HTTP/2 DATA frame during graceful shutdown Nov 17, 2026
@He-Pin
He-Pin requested a review from pjfanning November 17, 2026 17:42
@pjfanning pjfanning added this to the 2.1.0 milestone Nov 17, 2026

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks correct for the targeted case: when pullNextFrame closes the last stream, onAllStreamsClosed -> completeIfDone no longer sees the stale pre-transition WaitingForData, and the onAllDataFlushed after the transition completes the connection. A few points:

Commit/PR format. Commit 4c4a82413 has no body (AGENTS.md asks for Motivation/Modification/Result/Tests/References on non-trivial commits). In the PR description, References should be Refs #.../Fixes #.../None - <context> (the Netty link can go in the body text), and Tests should list the commands run and their results.

// last DATA frame of a stream. Don't report all data as flushed until the transition is done; `updateState` calls
// `onAllDataFlushed` afterwards if the new state has flushed everything.
private var transitionDepth = 0
def hasFlushedAllData: Boolean = transitionDepth == 0 && allDataFlushed(_state)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The underlying issue is that Http2StreamHandling calls onAllStreamsClosed synchronously from inside a multiplexer transition. Gating hasFlushedAllData works, but any other re-entrant logic in completeIfDone still runs mid-transition. Was deferring onAllStreamsClosed until the transition completes considered (similar to how deferredStreamToEnqueue already defers enqueueOutStream)? If you prefer to keep this approach, could the changed contract ("returns false while a transition is in progress") also be documented on the trait declaration at line 50, since that's what Http2Demux codes against?

private def updateState(transition: MultiplexerState => MultiplexerState): Unit = {
val oldState = _state
val newState = transition(_state)
transitionDepth += 1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A depth counter suggests nested updateState calls are supported, but a nested call's result would still be overwritten by the outer _state = newState (that hazard predates this PR, e.g. if a stream callback inside pullNextFrame does multiplexer.pushControlFrame(RstStreamFrame(...))). If nesting isn't expected, a Boolean inTransition flag would be clearer about intent; if it is possible, it probably needs separate handling.

network.expectDecodedHEADERS(3, endStream = true)
// The rest of stream 1 must be held back until the WINDOW_UPDATE arrives, and termination must not complete
// in the meantime.
network.expectNoBytes(100.millis)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expectNoBytes(100.millis) + terminated.isCompleted shouldBe false is a timing-based negative check: on a slow runner it can pass even if termination were about to complete early. The positive assertions after the WINDOW_UPDATE are the strong part. Fine to keep as a smoke check, just noting it isn't conclusive.

Motivation:
Closing the last stream can synchronously check connection completion before
its final frame has been pushed. A flow-control-blocked graceful response
must deliver all of its data and any trailing headers before termination.

Modification:
Preserve PJ Fanning's transition-depth completion guard and strengthened
window-aware test. Document the hasFlushedAllData observation contract on the
trait, and clarify that the counter does not make nested transitions safe.
Cover the same normal server lifecycle with and without response trailers.
Add ordinary streamed POST/trailer completion after an early final response.
Describe the short no-output assertion as a smoke check, not conclusive proof.

Result:
The final response bytes and optional END_STREAM trailing headers precede
connection completion. No change to nested-transition handling is claimed.

Tests:
- Offline Scala 2.13.18 production compilation with fatal warnings: passed.
- Extracted unchanged two-case server test block: two passed with cached
  HTTP/2 test harness and module classes, not a clean full build.
- Full Http2ServerSpec compilation unavailable: cached classpath lacks
  RequestBuilding and HPackSpecExamples.
- Extracted client test using existing scaffold: one passed with cached
  support classes; no baseline-failure claim for this additional coverage.
- Cached scalafmt 3.11.5 and git diff --check: passed.
- Full sbt, cross-Scala and aggregate MiMa: not run.

References:
Refs apache#1455 and PJ Fanning review 5946428716. Preserves PJ Fanning's code and
review improvements from the previous PR history.
@He-Pin
He-Pin force-pushed the test/http2-graceful-credit branch from dd6f192 to 9886649 Compare November 17, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants