Skip to content

S3: compute request checksums only when required #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sduchesneau merged 2 commits into from
Jul 9, 2026
Merged

S3: compute request checksums only when required #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
21c9a1b
S3: compute request checksums only when required
sduchesneau Jul 9, 2026
1221635
Merge branch 'develop' into fix/s3-request-checksum-when-required
sduchesneau Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

* S3 store: suppress checksum validation warnings from the SDK by setting `DisableLogOutputChecksumValidationSkipped` to `true`.

* S3 store: only compute request checksums when required (`RequestChecksumCalculation = WhenRequired`). Since `service/s3` v1.73.0 the SDK computes CRC32 checksums on PUT/multipart uploads by default, which breaks non-AWS S3-compatible backends that reject the parts. This removes the need for the `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` env var workaround. See https://github.com/aws/aws-sdk-go-v2/discussions/2960.

## v0.2.3

### Fixed
Expand Down
8 changes: 8 additions & 0 deletions s3store.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,14 @@ func newS3StoreContext(ctx context.Context, baseURL *url.URL, extension, compres
// Suppress flood of "Response has no supported checksum" warnings from the SDK.
// See https://github.com/aws/aws-sdk-go-v2/issues/3020
o.DisableLogOutputChecksumValidationSkipped = true

// Only compute request checksums when the operation requires it. As of
// service/s3 v1.73.0 the SDK computes CRC32 checksums on PUT/multipart by
// default, which breaks non-AWS S3-compatible backends (they reject the
// parts). This is the programmatic equivalent of setting the env var
// AWS_REQUEST_CHECKSUM_CALCULATION=when_required.
// See https://github.com/aws/aws-sdk-go-v2/discussions/2960
o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
})
s.uploader = manager.NewUploader(s.client)
s.downloader = manager.NewDownloader(s.client)
Expand Down
Loading