2026.7.11

UploadPartCopy passthrough — fixes s3proxy OOM during Scylla backup dedup (#115).

Problem: Dedup copies use UploadPartCopy on encrypted SSTables. The re-encrypt path downloaded, decrypted, and re-encrypted each ~1280MB object (~88MB peak per internal part × concurrent copies). With 13 agents and maxConnPerPod: 20, pods OOMKilled (exit 137) and backups showed Failed ≈ Deduplicated.

Fix:

• Passthrough for eligible copies: encrypted multipart sidecar source, full-object copy (no CopySourceRange), same credential — native server-side upload_part_copy of existing ciphertext with source DEK adoption (no decrypt/re-encrypt amplification)
• Hold governor reservation through upload_part completion (release after ciphertext is handed off)
• Default S3PROXY_MAX_PARALLEL_COPIES=2 pipeline semaphore per pod

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.11

Verify after rollout:

# During Scylla backup dedup: Failed should stop tracking Deduplicated
kubectl exec -n scylla-manager deploy/scylla-manager -- sctool progress <task-id> -c searchengine

# s3proxy OOMKills should drop; passthrough log lines appear
kubectl -n s3proxy-python logs deploy/s3proxy-python | grep UPLOAD_PART_COPY_PASSTHROUGH

Pre-merge gate:

make verify-passthrough

No config changes required. Encryption at rest unchanged — ciphertext is byte-copied, not decrypted to disk.

2026.7.10

Per-part copy memory reservation — fixes sustained 176MB governor hold during Scylla backup (#114).

Problem: #113 fixed O(1) peak sizing (~88MB per copy) but still reserved that amount for the entire copy duration (40 internal parts × ~24s for a 1280MB SSTable). Two concurrent copies held ~176MB continuously (active_mb=176.38), blocking a third and causing MEMORY_BACKPRESSURE even while INTERNAL_PART_UPLOADED events kept firing.

Fix: Streaming UploadPartCopy now acquires/releases governor memory per 32MB internal part (~88MB peak during encrypt only). Memory is free during S3 upload I/O between parts, so concurrent copies interleave within the 192MB budget.

Also: legacy encrypted objects without isec-kid now pass through byte-identically on COPY instead of failing re-encrypt.

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.10

Verify after rollout:

# Governor sawtooth during backup (not flat ~176MB)
kubectl -n s3proxy-python port-forward deploy/s3proxy-python 9090:9090 &
watch -n1 'curl -s localhost:9090/metrics | grep s3proxy_memory_reserved_bytes'

Or run pre-merge gate locally:

make verify-copy-memory

No config changes required. Keep S3PROXY_MEMORY_LIMIT_MB=192.

2026.7.9

Fix Scylla backup 503 storm: server-side copies are now O(1) memory (#113).

Problem: Scylla dedup UploadPartCopy of ~4.7GB SSTables sized internal parts as object_size/20 (~238MB), so each copy peaked at ~535MB — larger than the 192MB governor budget. The limiter clamped the reservation to the whole budget (exclusive slot), which could only be admitted when active_bytes == 0. Under steady backup traffic that never happened → endless MEMORY_BACKPRESSURE (requested_mb == limit_mb) → 503 SlowDown (~50% failure rate observed in prod).

Fix: Copies frame at a fixed 32MB internal part size (crypto.copy_internal_part_size), so copy peak is ~88MB regardless of object size. Copies fit the budget outright, never monopolize it, and run concurrently like normal requests. Large copies allocate internal parts sequentially (safe: prod copies are all partNumber=1, reassembly is metadata-driven).

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.9

Verify after rollout:

kubectl -n s3proxy-python exec deploy/s3proxy-python -- python3 -c "
from s3proxy import crypto; MB=1024**2
print('peak_4767MB:', round(crypto.copy_pipeline_peak(4767*MB)/MB,1), 'MB')
print('has copy_internal:', hasattr(crypto,'copy_internal_part_size'))
"

Expected: peak_4767MB: 88.0, has copy_internal: True

No config changes required. Keep S3PROXY_MEMORY_LIMIT_MB=192 — copies no longer need a higher budget.

2026.7.8

Fix copy starvation in the memory governor (#112).

• A budget-monopolizing server-side COPY (multi-GB manifest — reserves the whole governor budget to run exclusively) could be starved indefinitely by a steady stream of small requests: it backpressured (requested_mb == limit_mb) until it hit the timeout and 503'd.
• Add writer preference: while an exclusive copy is waiting, new non-exclusive requests are held back so active memory drains to 0 and the copy gets its slot (bounded by in-flight request lifetimes).
• Copies still reserve the whole budget and run one-at-a-time; the multi-copy OOM protection (#110) is unchanged.

No config or migration changes.

2026.7.7

Expose HAProxy Prometheus metrics on the bundled frontproxy (#111).

• Enable HAProxy's native /metrics endpoint on s3proxy-python-frontproxy (port 8404)
• Expose haproxy_backend_current_queue, haproxy_server_current_queue, haproxy_frontend_current_sessions, etc.
• Add a Prometheus Operator ServiceMonitor for Thanos scraping (configurable via frontproxy.metrics.*)

Backwards compatible (observability only; S3 traffic path unchanged). No migration.

2026.7.6

Fix copy-path memory governor under-reserve for multi-GB manifest copies (#110).

• Add copy_governor_clamped_reserve() so large server-side copies monopolize the governor budget instead of being crushed to the routine upload peak (~59MB)
• Route UploadPartCopy and CopyObject through reserve_copy_memory() instead of upload-style reserve_memory()
• Tighten copy_pipeline_peak() for large internal parts (add part // 7 slack for aiobotocore signing buffers on multi-GB manifests)
• Shard integration CI into parallel matrix jobs; update branch ruleset required checks accordingly

Backwards compatible (copy-path reservation logic only). No migration.

2026.7.5

Fix memory governor over-count and clamp starvation under Scylla backup load (#109).

• Tighten streaming_upload_peak to measured framed-upload peaks (4*part small, 2*part + FRAME large) instead of 2*part + 2*frame which scaled with multi-GB Content-Length
• Add governor_memory_footprint() to cap request-gate reservations at the 512MB-workload peak (~67MB) so a 6.6GB Content-Length no longer reserves 500MB+
• When an estimate exceeds the governor budget, clamp to the routine-workload peak — not the entire budget — so concurrent ~50MB Scylla parts are not starved behind one oversized slot

Backwards compatible (governor reservation logic only; upload path unchanged). No migration.

2026.7.4

Fix OOM in the streaming UploadPartCopy path under high copy concurrency (#108).

• Frame the copy writer like the UploadPart path: encrypt FRAME_PLAINTEXT_SIZE frames one internal part at a time; peak drops from ~192MB to ~32MB per copy, independent of object size.
• Size internal parts with memory_bounded_part_size (caps parts per client, fixes a latent >1.28GB range collision).
• copy_pipeline_peak now reserves the real framed peak; gate the previously ungoverned small-copy path.

Backwards compatible (writer-only; reader unchanged, legacy single-seal parts still decode). No migration.

2026.7.3

Features

• Server-side passthrough for same-credential encrypted COPY (#107). Scylla Manager's backup copies each SSTable to its snapshot-tagged key (CopyObject, MetadataDirective=COPY, same bucket). Every encrypted copy previously went through _copy_encrypted — GET source from upstream → decrypt → re-encrypt → PUT dest — turning a metadata-only "rename" into a full download + re-upload. Measured live this drove ~750 MB/s up and ~730 MB/s down to Hetzner (bytes_encrypted ≈ bytes_decrypted), saturating the CPU/AES-bound fleet and triggering a PutObject 503 SlowDown storm that stalled the daily backup at ~16%.

  The encryption is not key-bound (GCM AAD is None, the DEK is random and stored in the isec metadata / multipart sidecar, nonces are embedded in the ciphertext), so a byte-identical copy that keeps the same wrapped-DEK metadata decrypts under any key name. handle_copy_object now issues a native server-side CopyObject when the directive is COPY, the source was wrapped by the calling credential (re-key would be a no-op), and the ciphertext is ≤ 5 GiB; multipart objects also get their .meta sidecar server-side-copied. Cross-credential copies, REPLACE, and >5 GiB objects still take the decrypt/re-encrypt path.

  Verified against real Hetzner: CopyObject COPY on a 1.68 GB encrypted SSTable preserves isec/isec-kid metadata byte-for-byte.

Chart and image published as 2026.7.3.

2026.7.2

Fixes

• frontproxy: add option httpclose to stop HAProxy's built-in 400 Bad request on aborted multipart uploads (#106). When a backend pod is killed mid-body (KEDA churn + option redispatch), the response is sent before the request body is drained; on a kept-alive client connection the leftover body bytes were parsed as the next request → 400. Closing per response prevents reuse of a poisoned connection.

Chart and image published as 2026.7.2.