Repository navigation
Conversation
Motivation: During graceful shutdown, if the multiplexer was already pulled (WaitingForData) when a WINDOW_UPDATE released the last DATA frame of the last stream, sendDataFrame pulled the frame first. Pulling closed the stream, so completeIfDone saw hasFlushedAllData (still the old state) and completed the stage before the frame was pushed. The push then failed with "Cannot push closed port" and the final DATA frame was lost. Modification: Track multiplexer state transitions and report hasFlushedAllData as false while one is in progress. updateState already calls onAllDataFlushed once the new state has flushed everything. Result: The final DATA frame is sent before the connection completes. Tests: - sbt "http2-tests/testOnly ...Http2ServerSpec ...Http2ClientSpec": 195 passed, 17 pending - The tightened graceful shutdown test in this PR fails without this change References: Refs apache#1455
Motivation: The test could not tell whether the server held back stream data until the WINDOW_UPDATE arrived, and the probe's own stream window check could not fire because it assumed the default initial window. Modification: Tell the probe about the 8-byte stream window, use expectDATA instead of a manual loop, and require silence (with termination still pending) before sending the WINDOW_UPDATE. Result: The test fails if the server sends past its stream credit or completes shutdown early. Tests: - sbt "http2-tests/testOnly ...Http2ServerSpec ...Http2ClientSpec": 195 passed, 17 pending References: Refs apache#1455 review
c660cf6 to
dd6f192
Compare
|
I rebased this onto
This PR now contains a production fix, so the title and body may need updating. Feel free to revert anything you don't agree with or split it out. |
pjfanning
left a comment
There was a problem hiding this comment.
The fix looks correct for the targeted case: when pullNextFrame closes the last stream, onAllStreamsClosed -> completeIfDone no longer sees the stale pre-transition WaitingForData, and the onAllDataFlushed after the transition completes the connection. A few points:
Commit/PR format. Commit 4c4a82413 has no body (AGENTS.md asks for Motivation/Modification/Result/Tests/References on non-trivial commits). In the PR description, References should be Refs #.../Fixes #.../None - <context> (the Netty link can go in the body text), and Tests should list the commands run and their results.
| // last DATA frame of a stream. Don't report all data as flushed until the transition is done; `updateState` calls | ||
| // `onAllDataFlushed` afterwards if the new state has flushed everything. | ||
| private var transitionDepth = 0 | ||
| def hasFlushedAllData: Boolean = transitionDepth == 0 && allDataFlushed(_state) |
There was a problem hiding this comment.
The underlying issue is that Http2StreamHandling calls onAllStreamsClosed synchronously from inside a multiplexer transition. Gating hasFlushedAllData works, but any other re-entrant logic in completeIfDone still runs mid-transition. Was deferring onAllStreamsClosed until the transition completes considered (similar to how deferredStreamToEnqueue already defers enqueueOutStream)? If you prefer to keep this approach, could the changed contract ("returns false while a transition is in progress") also be documented on the trait declaration at line 50, since that's what Http2Demux codes against?
| private def updateState(transition: MultiplexerState => MultiplexerState): Unit = { | ||
| val oldState = _state | ||
| val newState = transition(_state) | ||
| transitionDepth += 1 |
There was a problem hiding this comment.
A depth counter suggests nested updateState calls are supported, but a nested call's result would still be overwritten by the outer _state = newState (that hazard predates this PR, e.g. if a stream callback inside pullNextFrame does multiplexer.pushControlFrame(RstStreamFrame(...))). If nesting isn't expected, a Boolean inTransition flag would be clearer about intent; if it is possible, it probably needs separate handling.
| network.expectDecodedHEADERS(3, endStream = true) | ||
| // The rest of stream 1 must be held back until the WINDOW_UPDATE arrives, and termination must not complete | ||
| // in the meantime. | ||
| network.expectNoBytes(100.millis) |
There was a problem hiding this comment.
expectNoBytes(100.millis) + terminated.isCompleted shouldBe false is a timing-based negative check: on a slow runner it can pass even if termination were about to complete early. The positive assertions after the WINDOW_UPDATE are the strong part. Fine to keep as a smoke check, just noting it isn't conclusive.
Motivation: Closing the last stream can synchronously check connection completion before its final frame has been pushed. A flow-control-blocked graceful response must deliver all of its data and any trailing headers before termination. Modification: Preserve PJ Fanning's transition-depth completion guard and strengthened window-aware test. Document the hasFlushedAllData observation contract on the trait, and clarify that the counter does not make nested transitions safe. Cover the same normal server lifecycle with and without response trailers. Add ordinary streamed POST/trailer completion after an early final response. Describe the short no-output assertion as a smoke check, not conclusive proof. Result: The final response bytes and optional END_STREAM trailing headers precede connection completion. No change to nested-transition handling is claimed. Tests: - Offline Scala 2.13.18 production compilation with fatal warnings: passed. - Extracted unchanged two-case server test block: two passed with cached HTTP/2 test harness and module classes, not a clean full build. - Full Http2ServerSpec compilation unavailable: cached classpath lacks RequestBuilding and HPackSpecExamples. - Extracted client test using existing scaffold: one passed with cached support classes; no baseline-failure claim for this additional coverage. - Cached scalafmt 3.11.5 and git diff --check: passed. - Full sbt, cross-Scala and aggregate MiMa: not run. References: Refs apache#1455 and PJ Fanning review 5946428716. Preserves PJ Fanning's code and review improvements from the previous PR history.
dd6f192 to
9886649
Compare
Motivation
Graceful HTTP/2 shutdown must deliver the final response DATA and any trailing HEADERS before completing the connection. A synchronous completion callback must not observe stale flushed state during a multiplexer transition.
Modification
Preserve PJ Fanning's production completion guard from 66ef4e5 and strengthened test from dd6f192. Document that hasFlushedAllData returns false while a transition is in progress. The depth counter guards observation only; it does not make nested state updates safe. Add normal server coverage with and without trailers and client streamed-request/trailer completion after an early final response.
Result
The final DATA and optional END_STREAM trailing HEADERS precede connection completion. This concerns stream-stage completion, not operating-system flush guarantees. The alternative notification design and nested-update questions remain open.
Tests
The short no-output assertion is only a smoke check; positive byte, trailer and completion assertions provide the main coverage.
References
Refs #1455.
Preserves PJ Fanning's contributions from 66ef4e5 and dd6f192.
Netty #18126 inspired ordinary lifecycle coverage; the Pekko implementation is independent.