fix: stream large unsigned PUTs once (deferred SigV4)#116
Merged
Conversation
Defer SigV4 verification for large bodies missing x-amz-content-sha256 so Scylla SST uploads hash while streaming instead of double-buffering in RAM. Add uvicorn limit_concurrency (S3PROXY_MAX_IN_FLIGHT) to bound httptools socket buffers outside the memory governor, abort multipart on client disconnect, and route the Docker entrypoint through main.py so limits apply. Co-authored-by: Cursor <cursoragent@cursor.com>
Use `is True` / isinstance(bytes) for deferred-sig and preloaded-body so MagicMock request.state does not trigger deferred verify or buffer errors in integration tests. Rename ClientDisconnectError for N818; treat only explicit True from is_disconnected() as a disconnect. Co-authored-by: Cursor <cursoragent@cursor.com>
Auto-derived limit_concurrency (memory_limit_mb // 8) rejected connections before the memory governor could queue them, breaking backpressure integration tests and causing 503/partial-upload SHA256 mismatches. Default 0 is now unlimited; set S3PROXY_MAX_IN_FLIGHT explicitly in prod when needed. Co-authored-by: Cursor <cursoragent@cursor.com>
track_chunk() called request.is_disconnected() every 8MB while reading the upload body. Starlette can dequeue http.request messages during that check and drop body bytes, corrupting SHA256 on large signed PUTs. Remove the polling path; rely on ClientDisconnect from request.stream() and abort multipart on disconnect. S3PROXY_MAX_IN_FLIGHT stays opt-in only (0 = unlimited, unchanged from main). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Not required for the OOM fix; memory governor already handles backpressure. Drops config, chart wiring, tests, and uvicorn limit_concurrency plumbing. Co-authored-by: Cursor <cursoragent@cursor.com>
ServerSideHannes
changed the title
Makes _parse_header_auth() return a named type instead of an opaque 8-element tuple, improving readability at all three call sites. Co-authored-by: Cursor <cursoragent@cursor.com>
2 tasks
ServerSideHannes
added a commit
that referenced
this pull request
Jul 8, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
x-amz-content-sha256no longer callrequest.body()before streaming — credentials are checked up front, body is hashed once during encrypt/upload, SigV4 verified before complete._iter_request_bodyavoids double-buffer when a small body was preloaded for signature (request.body()+request.stream()).python -m s3proxy.mainso TLS settings fromSettingsapply in prod.Not included
S3PROXY_MAX_IN_FLIGHT— memory governor (S3PROXY_MEMORY_LIMIT_MB) already handles backpressure.is_disconnected()during streaming (Starlette can drop body bytes and corrupt SHA256).Test plan
uv run pytest tests/unit/ -q— 534 passedtest_scylla_deferred_put_memory.py— 179MB deferred PUT peaks ~33MB, not ~358MB double-buffersignature_deferred(notbody_loaded~179MB); OOM rate drops on server pods