Skip to content

Support running in background - #72

Merged
jamesbornholt merged 1 commit into
mainfrom
running_in_background
Feb 16, 2023
Merged

Support running in background#72
jamesbornholt merged 1 commit into
mainfrom
running_in_background

Conversation

@monthonk

@monthonk monthonk commented Feb 7, 2023

Copy link
Copy Markdown
Contributor

This change allows the file connector to run in background and it will be running there by default. Users can change this behavior by passing in mount option --foreground or -f to run it in foreground instead.

I'm still figuring out how we can create some tests for this change. Any ideas are welcome. Another thing is that the logs are currently writing to the stdout directly even if the process is running in background. It's a bit annoying but I think that would be a part of PR that addresses #39.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@monthonk
monthonk force-pushed the running_in_background branch from 3939a18 to 147d0bd Compare February 7, 2023 16:23
@dannycjones
dannycjones self-requested a review February 8, 2023 18:04
@monthonk
monthonk force-pushed the running_in_background branch 3 times, most recently from e308c2f to 9b7b556 Compare February 9, 2023 14:14
@monthonk

monthonk commented Feb 9, 2023

Copy link
Copy Markdown
Contributor Author

From this PR, I think AutoUnmount should be set as a default config, otherwise the mount state would be left hanging when we kill the background process.

@monthonk
monthonk force-pushed the running_in_background branch 4 times, most recently from 707c137 to 56f2f9e Compare February 9, 2023 16:39
@monthonk

monthonk commented Feb 9, 2023

Copy link
Copy Markdown
Contributor Author

ASan is not happy with the tests. Any thoughts?

dannycjones
dannycjones reviewed Feb 9, 2023
View reviewed changes
Comment thread s3-file-connector/Cargo.toml Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/tests/common/mod.rs Outdated
Comment thread s3-file-connector/tests/cli.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
jamesbornholt
jamesbornholt reviewed Feb 9, 2023
View reviewed changes
Comment thread s3-file-connector/tests/cli.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/tests/cli.rs Outdated
Comment thread s3-file-connector/tests/cli.rs Outdated
Comment thread s3-file-connector/tests/cli.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
@monthonk
monthonk force-pushed the running_in_background branch 11 times, most recently from 131cb22 to 9817ad8 Compare February 13, 2023 10:08
monthonk
monthonk commented Feb 13, 2023
View reviewed changes
Comment thread s3-file-connector/src/main.rs Outdated
monthonk
monthonk commented Feb 13, 2023
View reviewed changes
Comment thread Makefile Outdated
@monthonk
monthonk force-pushed the running_in_background branch from 9817ad8 to 43703f5 Compare February 13, 2023 11:12
jamesbornholt
jamesbornholt reviewed Feb 13, 2023
View reviewed changes
Comment thread Makefile Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment on lines 155 to 162

@jamesbornholt jamesbornholt Feb 13, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kind of subtle and we should be more explicit -- the session stays running because its lifetime is bound to the match statement, so doesn't drop until after the park. Probably we should write Ok(_session) and put a comment above the park explaining this.

Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/tests/common/mod.rs Outdated
Comment thread s3-file-connector/tests/fuse_tests/fork_test.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
@monthonk
monthonk force-pushed the running_in_background branch from fdc740d to 88d87ab Compare February 13, 2023 17:59
jamesbornholt
jamesbornholt reviewed Feb 13, 2023
View reviewed changes
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated

@jamesbornholt jamesbornholt Feb 13, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yeah, this is unlikely but we should do it properly. That means creating a pipe (pipe2) before the fork, writing a character into the write side of the pipe when the child wants to send its status (probably File::from_raw_fd), and reading a character off the read side of the pipe in the parent to check the status (same). Doing timeouts will be more annoying this way, since you can't configure a timeout for file/pipe reads. Might need to spawn a thread to do it.

@monthonk
monthonk force-pushed the running_in_background branch from 88d87ab to d6dc79c Compare February 14, 2023 10:04
dannycjones
dannycjones requested changes Feb 14, 2023
View reviewed changes

@dannycjones dannycjones left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little uneasy on some of the multi-process logic, not that I understand the area well at this point.

Also, I expect the logic to change anyway with the pipe change.

Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/tests/fuse_tests/fork_test.rs Outdated
@monthonk
monthonk force-pushed the running_in_background branch 5 times, most recently from 00ac0ea to 67cedc7 Compare February 15, 2023 14:48
@monthonk
monthonk force-pushed the running_in_background branch from 67cedc7 to 6e52bb5 Compare February 15, 2023 16:22
dannycjones
dannycjones reviewed Feb 15, 2023
View reviewed changes
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment on lines 177 to 182

@dannycjones dannycjones Feb 15, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need some kind of stopping condition.

When we receive SIGINT, we should exit. Maybe another thread waits for SIGINT, and sets an atomic bool?

Does this sound right, @jamesbornholt?

@monthonk monthonk Feb 15, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the standard library already handle that. I have tested this by running kill -2 {pid} which send SIGINT to the running child process and it works just fine.

@jamesbornholt jamesbornholt Feb 15, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to figure something different out here, otherwise the process runs forever even if the user unmounts it. But we can do that as a followup, I think, because we'll probably have to make fuser changes.

@monthonk monthonk Feb 16, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that is existing problem, it keep running forever even for foreground process. It's just less visible when run in the background. Let's do it as a follow up.

@dannycjones dannycjones Feb 16, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's open an issue and then we can resolve this.

@monthonk monthonk Feb 16, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #93.

Comment thread s3-file-connector/src/main.rs Outdated
@monthonk
monthonk force-pushed the running_in_background branch from 6e52bb5 to d465085 Compare February 15, 2023 18:19
jamesbornholt
jamesbornholt reviewed Feb 15, 2023
View reviewed changes
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment on lines 177 to 182

@jamesbornholt jamesbornholt Feb 15, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to figure something different out here, otherwise the process runs forever even if the user unmounts it. But we can do that as a followup, I think, because we'll probably have to make fuser changes.

Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment thread s3-file-connector/src/main.rs Outdated
Comment on lines 198 to 199

@jamesbornholt jamesbornholt Feb 15, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe do this conversion outside the thread and right below closing the write_fd, to make clearer the fate of the pipe (and give f a better name)

@monthonk
Support running in background
17b1af2 
This change allows the file connector to run in background and it will be running there by default.
Users can change this behavior by passing in mount option `--foreground` or `-f` to run it in foreground instead.

Signed-off-by: Monthon Klongklaew <monthonk@amazon.co.uk>
@monthonk
monthonk force-pushed the running_in_background branch from d465085 to 17b1af2 Compare February 16, 2023 10:47
dannycjones
dannycjones approved these changes Feb 16, 2023
View reviewed changes

@dannycjones dannycjones left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending new issue to follow up on failing to exit if unmounted.

@jamesbornholt
jamesbornholt merged commit f6e936f into main Feb 16, 2023
@jamesbornholt
jamesbornholt deleted the running_in_background branch February 16, 2023 15:42
@monthonk monthonk mentioned this pull request Feb 16, 2023
Merged
monthonk added a commit that referenced this pull request Feb 16, 2023
@monthonk
Fix benchmark script
69da683 
After #72, our file connector will be running in background by default.
This breaks the benchmark clean up process because we're now using the
wrong pid to kill the process after unmount.

Technically, running in background should not be a problem but because
of #93 we will have to use `--foreground` flag until we have it fixed.

Signed-off-by: Monthon Klongklaew <monthonk@amazon.co.uk>
monthonk added a commit that referenced this pull request Feb 16, 2023
@monthonk
Fix benchmark script (#94)
1d03fdc 
* Fix benchmark script
After #72, our file connector will be running in background by default.
This breaks the benchmark clean up process because we're now using the
wrong pid to kill the process after unmount.

Technically, running in background should not be a problem but because
of #93 we will have to use `--foreground` flag until we have it fixed.

Signed-off-by: Monthon Klongklaew <monthonk@amazon.co.uk>

* Update file system name in bench script

Signed-off-by: Monthon Klongklaew <monthonk@amazon.co.uk>

---------

Signed-off-by: Monthon Klongklaew <monthonk@amazon.co.uk>
passaro added a commit to passaro/mountpoint-s3 that referenced this pull request Feb 5, 2025
@passaro
Update CRT submodules to latest releases
4936e9f 
Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b:
  > A bunch of CMake fixes (awslabs#258)
  > Add Account Id to Credentials (awslabs#260)
  > Skip Transfer-Encoding from signing (awslabs#261)
Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab:
  > Fix Findcrypto.cmake (awslabs#205)
  > A bunch of CMake fixes (awslabs#203)
  > Switch CI to use roles (awslabs#202)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa:
  > A bunch of CMake fixes (awslabs#1178)
  > Fix heap overflow on uri parsing (awslabs#1185)
  > (take 2) Detect when AVX is disabled via OSXSAVE (awslabs#1184)
  > Fixup IPv6 validation logic (awslabs#1180)
  > Detect when AVX is disabled via OSXSAVE (awslabs#1182)
  > proof_ci.yaml must use latest upload-artifact (awslabs#1183)
  > change PR template to ask for clearer wording (awslabs#1177)
Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b:
  > A bunch of CMake fixes (awslabs#72)
  > Switch CI to use roles (awslabs#71)
  > chore: Modified bug issue template to add checkbox to report potential regression. (awslabs#69)
Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59:
  > A bunch of CMake fixes (awslabs#497)
  > Fix CI for GCC-13 on Ubuntu-18  (awslabs#496)
  > Switch CI to use roles (awslabs#494)
Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf:
  > A bunch of CMake fixes (awslabs#701)
  > Event Loop & Socket Type Multi-Support (awslabs#692)
  > fix typo in log message (awslabs#702)
  > Fix CI for GCC-13 on Ubuntu-18 (awslabs#700)
  > Switch CI to use roles (awslabs#698)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53:
  > A bunch of CMake fixes (awslabs#480)
  > S3Express CreateSession Allowlist Headers (awslabs#492)
  > Auto - Update S3 Ruleset & Partition (awslabs#491)
Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa:
  > A bunch of CMake fixes (awslabs#50)
Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8:
  > A bunch of CMake fixes (awslabs#101)
  > Switch CI to use roles (awslabs#100)
Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3:
  > Prepare AWS-LC v1.44.0 (#2153)
  > Fix issue with ML-DSA key parsing (#2152)
  > Add support for PKCS7_set/get_detached (#2134)
  > Prepare Docker image for CI integration jobs (#2126)
  > Delete OpenVPN mainline patch from our integration build (#2149)
  > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101)
  > Support keypair calculation for PQDSA PKEY (#2145)
  > Optimize x86/aarch64 MD5 implementation (#2137)
  > Check for MIPSEB in target.h (#2143)
  > Ed25519ph and Ed25519ctx Support (#2120)
  > Support for ML-DSA public key generation from private key (#2142)
  > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140)
  > Remove remaining support for Trusty and Fuchsia operating systems (#2136)
  > ACVP test harness for ML-DSA (#2127)
  > Minor symbols to work with Ruby's mainline (#2132)

Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
github-merge-queue Bot pushed a commit that referenced this pull request Feb 5, 2025
@passaro
Update CRT submodules to latest releases (#1255)
2bf0385 
Update the CRT libraries to the latest releases. In particular, include:
* S3Express CreateSession Allowlist Headers
([awslabs/aws-c-s3#492](awslabs/aws-c-s3#492))

<details>
  <summary>Full CRT changelog:</summary>
  
```
Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b:
  > A bunch of CMake fixes (#258)
  > Add Account Id to Credentials (#260)
  > Skip Transfer-Encoding from signing (#261)
Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab:
  > Fix Findcrypto.cmake (#205)
  > A bunch of CMake fixes (#203)
  > Switch CI to use roles (#202)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa:
  > A bunch of CMake fixes (#1178)
  > Fix heap overflow on uri parsing (#1185)
  > (take 2) Detect when AVX is disabled via OSXSAVE (#1184)
  > Fixup IPv6 validation logic (#1180)
  > Detect when AVX is disabled via OSXSAVE (#1182)
  > proof_ci.yaml must use latest upload-artifact (#1183)
  > change PR template to ask for clearer wording (#1177)
Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b:
  > A bunch of CMake fixes (#72)
  > Switch CI to use roles (#71)
  > chore: Modified bug issue template to add checkbox to report potential regression. (#69)
Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59:
  > A bunch of CMake fixes (#497)
  > Fix CI for GCC-13 on Ubuntu-18  (#496)
  > Switch CI to use roles (#494)
Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf:
  > A bunch of CMake fixes (#701)
  > Event Loop & Socket Type Multi-Support (#692)
  > fix typo in log message (#702)
  > Fix CI for GCC-13 on Ubuntu-18 (#700)
  > Switch CI to use roles (#698)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53:
  > A bunch of CMake fixes (#480)
  > S3Express CreateSession Allowlist Headers (#492)
  > Auto - Update S3 Ruleset & Partition (#491)
Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa:
  > A bunch of CMake fixes (#50)
Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8:
  > A bunch of CMake fixes (#101)
  > Switch CI to use roles (#100)
Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3:
  > Prepare AWS-LC v1.44.0 (#2153)
  > Fix issue with ML-DSA key parsing (#2152)
  > Add support for PKCS7_set/get_detached (#2134)
  > Prepare Docker image for CI integration jobs (#2126)
  > Delete OpenVPN mainline patch from our integration build (#2149)
  > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101)
  > Support keypair calculation for PQDSA PKEY (#2145)
  > Optimize x86/aarch64 MD5 implementation (#2137)
  > Check for MIPSEB in target.h (#2143)
  > Ed25519ph and Ed25519ctx Support (#2120)
  > Support for ML-DSA public key generation from private key (#2142)
  > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140)
  > Remove remaining support for Trusty and Fuchsia operating systems (#2136)
  > ACVP test harness for ML-DSA (#2127)
  > Minor symbols to work with Ruby's mainline (#2132)
```
</details>


### Does this change impact existing behavior?

No.

### Does this change need a changelog entry? Does it require a version
change?

No.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
mansi153 pushed a commit to mansi153/mountpoint-s3 that referenced this pull request Jul 24, 2025
@passaro @mansi153
Update CRT submodules to latest releases (awslabs#1255)
12fe83b 
Update the CRT libraries to the latest releases. In particular, include:
* S3Express CreateSession Allowlist Headers
([awslabs/aws-c-s3#492](awslabs/aws-c-s3#492))

<details>
  <summary>Full CRT changelog:</summary>
  
```
Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 5bc67797..b513db4b:
  > A bunch of CMake fixes (awslabs#258)
  > Add Account Id to Credentials (awslabs#260)
  > Skip Transfer-Encoding from signing (awslabs#261)
Submodule mountpoint-s3-crt-sys/crt/aws-c-cal fbbe2612..7299c6ab:
  > Fix Findcrypto.cmake (awslabs#205)
  > A bunch of CMake fixes (awslabs#203)
  > Switch CI to use roles (awslabs#202)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 7a6f5df2..0e7637fa:
  > A bunch of CMake fixes (awslabs#1178)
  > Fix heap overflow on uri parsing (awslabs#1185)
  > (take 2) Detect when AVX is disabled via OSXSAVE (awslabs#1184)
  > Fixup IPv6 validation logic (awslabs#1180)
  > Detect when AVX is disabled via OSXSAVE (awslabs#1182)
  > proof_ci.yaml must use latest upload-artifact (awslabs#1183)
  > change PR template to ask for clearer wording (awslabs#1177)
Submodule mountpoint-s3-crt-sys/crt/aws-c-compression c6c1191e..f951ab2b:
  > A bunch of CMake fixes (awslabs#72)
  > Switch CI to use roles (awslabs#71)
  > chore: Modified bug issue template to add checkbox to report potential regression. (awslabs#69)
Submodule mountpoint-s3-crt-sys/crt/aws-c-http fc3eded2..590c7b59:
  > A bunch of CMake fixes (awslabs#497)
  > Fix CI for GCC-13 on Ubuntu-18  (awslabs#496)
  > Switch CI to use roles (awslabs#494)
Submodule mountpoint-s3-crt-sys/crt/aws-c-io fcb38c80..3041dabf:
  > A bunch of CMake fixes (awslabs#701)
  > Event Loop & Socket Type Multi-Support (awslabs#692)
  > fix typo in log message (awslabs#702)
  > Fix CI for GCC-13 on Ubuntu-18 (awslabs#700)
  > Switch CI to use roles (awslabs#698)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 a3b401bf..6eb8be53:
  > A bunch of CMake fixes (awslabs#480)
  > S3Express CreateSession Allowlist Headers (awslabs#492)
  > Auto - Update S3 Ruleset & Partition (awslabs#491)
Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils 1ae8664f..ba6a28fa:
  > A bunch of CMake fixes (awslabs#50)
Submodule mountpoint-s3-crt-sys/crt/aws-checksums 3e4101b9..fb8bd0b8:
  > A bunch of CMake fixes (awslabs#101)
  > Switch CI to use roles (awslabs#100)
Submodule mountpoint-s3-crt-sys/crt/aws-lc ffd6fb71..138a6ad3:
  > Prepare AWS-LC v1.44.0 (#2153)
  > Fix issue with ML-DSA key parsing (#2152)
  > Add support for PKCS7_set/get_detached (#2134)
  > Prepare Docker image for CI integration jobs (#2126)
  > Delete OpenVPN mainline patch from our integration build (#2149)
  > SHA3/SHAKE Init Updates via FIPS202 API layer (#2101)
  > Support keypair calculation for PQDSA PKEY (#2145)
  > Optimize x86/aarch64 MD5 implementation (#2137)
  > Check for MIPSEB in target.h (#2143)
  > Ed25519ph and Ed25519ctx Support (#2120)
  > Support for ML-DSA public key generation from private key (#2142)
  > Avoid mixing SSE and AVX in XTS-mode AVX512 implementation (#2140)
  > Remove remaining support for Trusty and Fuchsia operating systems (#2136)
  > ACVP test harness for ML-DSA (#2127)
  > Minor symbols to work with Ruby's mainline (#2132)
```
</details>


### Does this change impact existing behavior?

No.

### Does this change need a changelog entry? Does it require a version
change?

No.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
yerzhan7 added a commit to yerzhan7/mountpoint-s3 that referenced this pull request Sep 22, 2026
@yerzhan7
Update CRT submodules to latest releases (awslabs#1967)
ffa5658 
Update the CRT submodules to the latest releases:

* `aws-c-auth` `v0.10.4` -> `v1.0.0`
* `aws-c-cal` `v0.9.14` -> `v1.0.0`
* `aws-c-common` `v0.14.2` -> `v1.0.1`
* `aws-c-compression` `v0.3.2` -> `v1.0.0`
* `aws-c-http` `v0.11.0` -> `v1.0.0`
* `aws-c-io` `v0.27.3` -> `v1.0.0`
* `aws-c-s3` `v0.12.8` -> `v1.1.3`
* `aws-c-sdkutils` `v0.2.7` -> `v1.0.0`
* `aws-checksums` `v0.2.10` -> `v1.0.0`
* `aws-lc` `v5.1.0` -> `v5.9.0`
* `s2n-tls` `v1.7.5` -> `v1.7.10`

Also exclude aws-lc's new Keccak test vectors from the published
mountpoint-s3-crt-sys archive, as they are not required to build the
CRT. The compressed crate is 9.2MiB, within the 10MiB crates.io limit.

<details>
  <summary>Full CRT changelog:</summary>

```
Submodule mountpoint-s3-crt-sys/crt/aws-c-auth 4b5d524bf..055e822f0:
  > chore(release): add governance files (awslabs#303)
  > Clean up docs (awslabs#301)
  > chore(release): patch-update to VERSION - 0.10.5
  > CI/CD improvements (awslabs#302)
  > Use latest as aws-crt-builder version (awslabs#300)
  > Add thread sanitizer CI job (awslabs#299)
Submodule mountpoint-s3-crt-sys/crt/aws-c-cal 9edd8eac2..2441187f3:
  > chore(release): add governance files (awslabs#257)
  > Clean up docs (awslabs#255)
  > CI/CD improvements (awslabs#256)
  > Expand ecc validation test (awslabs#254)
  > ECC Validation Test (awslabs#253)
  > Use latest as aws-crt-builder version (awslabs#252)
  > Swap fatal for soft error (awslabs#251)
  > Update deprecated OpenBSD CI job (awslabs#250)
  > Regression Labeler Fix (awslabs#249)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common a9d57d2d3..69cd45209:
  > chore(release): patch-update to VERSION - 1.0.1
  > Helpers for file descriptor opertaions.  (awslabs#1272)
  > Debug API to override default thread options (awslabs#1273)
  > Bullseye -> Bookworm (awslabs#1276)
  > Fix wrong libdir on some platforms (awslabs#1271)
  > Add some error code utility functions (awslabs#1270)
  > External shutdown error (awslabs#1269)
  > chore(release): add governance files (awslabs#1267)
  > Suppress MSAN false positive (awslabs#1264)
  > Clean up docs (awslabs#1263)
  > Release 0.14.5 (patch)
  > CI/CD improvements (awslabs#1250)
  > Use latest as aws-crt-builder version (awslabs#1261)
  > aws_file_get_last_modified_epoch (awslabs#1258)
  > Fix windows nonascii env (awslabs#1260)
  > Add HWASan suppression support (awslabs#1256)
Submodule mountpoint-s3-crt-sys/crt/aws-c-compression d8264e64f..891312c05:
  > chore(release): add governance files (awslabs#81)
  > Clean up docs (awslabs#79)
  > chore(release): patch-update to VERSION - 0.3.3
  > CI/CD improvements (awslabs#80)
  > Use latest as aws-crt-builder version (awslabs#78)
  > Regression Labeler Fix (awslabs#77)
Submodule mountpoint-s3-crt-sys/crt/aws-c-http 8aefd899f..2b563f8a7:
  > chore(release): add governance files (awslabs#575)
  > Clean up docs (awslabs#573)
  > chore(release): patch-update to VERSION - 0.11.1
  > CI/CD improvements (awslabs#574)
  > HPACK String Length Validation (awslabs#572)
  > Enforce MAX_HEADER_LIST_SIZE (awslabs#568)
  > Fix use-after-free in stats gathering during connection shutdown(awslabs#570)
  > Use latest as aws-crt-builder version (awslabs#569)
  > Regression Labeler Fix (awslabs#564)
  > Support s2n-tls on macOS (awslabs#560)
Submodule mountpoint-s3-crt-sys/crt/aws-c-io 8bda5cf0f..1685abcd3:
  > chore(release): add governance files (awslabs#828)
  > Clean up docs (awslabs#821)
  > chore(release): patch-update to VERSION - 0.27.7
  > CI/CD improvements (awslabs#824)
  > Fix read-window stall by gating window updates on the scarcest slot (awslabs#818)
  > Automate test cert renewal (awslabs#820)
  > Use latest as aws-crt-builder version (awslabs#817)
  > Expose option to set TCP_NODELAY on sockets (awslabs#816)
  > log the negotiated TLS version (awslabs#815)
  > Handle optional s2n-tls in the install CMake config (awslabs#814)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 448ec5e49..805ca0f4c:
  > chore(release): patch-update to VERSION - 1.1.3
  > feat: add more metrics to S3 metrics object (awslabs#673)
  > Checksum validation handling cleanup (awslabs#678)
  > API to Get Default S3 Memory Limits (awslabs#681)
  > chore(release): patch-update to VERSION - 1.1.2
  > move the check and relax it for the file streaming case (awslabs#680)
  > chore(release): patch-update to VERSION - 1.1.1
  > Add S3 Metrics (awslabs#674)
  > chore(release): minor-update to VERSION - 1.1.0
  > Add retry_config to aws_s3_client_config (awslabs#672)
  > Retry on 502 504 errors (awslabs#671)
  > Sub 10Gbps S3 Throughput Handling (awslabs#670)
  > chore(release): add governance files (awslabs#668)
  > chore(release): patch-update to VERSION - 0.13.7
  > update doc on endpoint resolver update script (awslabs#667)
  > fix(copy-object): forward request-payer to the source-size HEAD (awslabs#654)
  > Clean up docs (awslabs#665)
  > chore(release): patch-update to VERSION - 0.13.6
  > CI/CD improvements (awslabs#666)
  > Combine per-part CRCs for whole-object download checksum validation (awslabs#663)
  > support opaque etags (awslabs#664)
  > fix resume token build (awslabs#661)
  > Use latest as aws-crt-builder version (awslabs#660)
  > [feat.] async pause and on_error_resume_token (awslabs#649)
  > Turn nagle off (awslabs#659)
  > Restore enums values that were deleted on accident. (awslabs#658)
  > fix ordering of read from stream (awslabs#655)
Submodule mountpoint-s3-crt-sys/crt/aws-c-sdkutils cb14fea36..de0fbe807:
  > chore(release): add governance files (awslabs#74)
  > Clean up docs (awslabs#72)
  > chore(release): patch-update to VERSION - 0.2.10
  > CI/CD improvements (awslabs#73)
  > Relax handling of unmodeled context params in bdd (awslabs#71)
  > Use latest as aws-crt-builder version (awslabs#70)
  > fix bdd header encoding (awslabs#69)
Submodule mountpoint-s3-crt-sys/crt/aws-checksums 1d5f2f1f3..ee7c435de:
  > chore(release): add governance files (awslabs#119)
  > Clean up docs (awslabs#116)
  > chore(release): patch-update to VERSION - 0.2.11
  > CI/CD improvements (awslabs#117)
  > Use latest as aws-crt-builder version (awslabs#115)
  > Update deprecated OpenBSD CI job (awslabs#114)
  > Regression Labeler Fix (awslabs#113)
  > Fix clang-cl compilation (awslabs#112)
Submodule mountpoint-s3-crt-sys/crt/aws-lc 6283365b1..39b142ec3:
  > Prepare v5.9.0 (#3523)
  > Report unexpected EOF as `SSL_ERROR_SSL` for OpenSSL 3.x parity (#3484)
  > Parallelise the serial Windows builds in Windows Alternative Compilers (#3516)
  > Avoid duplicate formal-verification runs on PR branches (#3526)
  > Fail the ABI check when abidiff cannot run (#3511)
  > Bump the pip-ci group across 1 directory with 3 updates (#3399)
  > Replace the error-queue trim helpers with a suppression scope (#3529)
  > Run the macOS jobs on a single OS version (#3513)
  > Stop building the test suite in the ABI diff image (#3512)
  > Thin the compiler-tests config axis (#3515)
  > Remove duplicate Python CRT integration work (#3525)
  > ci: enable shared libraries for clang-tidy provider build (#3520)
  > ci: fix OpenSSH integration (#3519)
  > ci: fix librelp integration test (#3517)
  > ci: fix mariadb integration (#3518)
  > Add ERR_num_errors and ERR_pop_to_count (#3501)
  > Cache the CMake-from-source image layers in CMake compatibility CI (#3510)
  > Add OpenSSL-compatible EVP_CTRL_CCM_* aliases (#3490)
  > Fix EVP signing-context control collision (#3458)
  > Fix BN_CTX leak on allocation failure in `EC_GROUP_new_curve_GFp` (#3506)
  > ci: handle expired Bullseye repository in legacy GCC jobs (#3508)
  > Export OpenSSL-compatible ECPKParameters buffer APIs (#3493)
  > Fix tpm2-tss integration build without SM4 (#3507)
  > Add legacy SSL function codes for OpenSSL compatibility (#3491)
  > Register -tls1_2 and -tls1_3 flags in bssl client (#3492)
  > Gate ruby master rubygems ML-DSA tests on key loading, not generation (#3479)
  > Use OpenSSL-compatible long names for ML-DSA OIDs (#3481)
  > Document the AWS-LC provider (#3434)
  > Implement SHA-256 through the AWS-LC provider (#3433)
  > ci: record integration failure metadata (#3482)
  > ci: fix failing librelp integration (#3480)
  > Add ML-KEM support to HPKE (draft-ietf-hpke-pq-05) (#3277)
  > Limit push-triggered CI to main and fips-* branches (#3478)
  > Prepare 5.8.0 (#3476)
  > Implement the AWS-LC provider interface (#3432)
  > Route req -extensions to the certificate and add -reqexts for the CSR (#3469)
  > Add FIPS_module_name to report the cryptographic module name (#3473)
  > Print FIPS module name in cli tooling (#3471)
  > Prepare 5.7.0 (#3465)
  > Fix EVP_DecryptUpdate output handling (#3460)
  > Define AT_HWCAP* when <sys/auxv.h> hides them (#3429)
  > Add SECURITY.md with AWS-LC threat model (#3421)
  > Prepare 5.6.0 (#3448)
  > Re-import mlkem-native and mldsa-native; drop custom meta headers (#3367)
  > Define BN_FLG_CONSTTIME as zero for source compatibility (#3446)
  > ci: consolidate android-omnibus deployments on external PRs (#3447)
  > ci: stop deployment noise for maintainers (#3445)
  > Install OpenSSL-compat libcrypto.pc and libssl.pc with the shim (#3370)
  > Pin librelp to a release, wrap `SSL_get_shutdown` for failing integration test (#3431)
  > Fix ML-DSA externalMu JSON tags (#3439)
  > Upgrade JDK to JDK21 for ACCP tests (#3440)
  > Decouple symbol versioning from distribution packaging mode (#3426)
  > Bump the cargo-ci-lambda group in /tests/ci/lambda with 4 updates (#3390)
  > Add latest compiler coverage to CI (#3406)
  > Move network-dependent s_client tests into integration_test (#3407)
  > Update pull request template (#3436)
  > Add AWS-LC provider build and CI infrastructure (#3419)
  > Add backport apply command (#3414)
  > ci: re-enable gcc-14 + FIPS build in gcc-14-hardened job (#3428)
  > ci: use preinstalled Rust in pyopenssl integration (#3430)
  > Fix Windows OPENSSL_SMALL Debug builds with NASM 2.16.01 (#3420)
  > ci: install Rust in the Linux docker images (#3393)
  > Enable mtr retries in MariaDB CI integration (#3427)
  > Add Backport Tool Analysis (#3389)
  > Reject GCM IV lengths below 8 in EVP_CTRL_GCM_IV_GEN (#3424)
  > ci: pin security review to the PR head commit (#3425)
  > Add Keccak-256 (Ethereum-style, original 0x01 padding) (#3245)
  > Add memcached integration test (#3410)
  > Add CPython 3.15 integration test (#3422)
  > Add brainpool EC_group symbols; make public API symbol registration self-service (#3423)
  > Harden CLI input validation and file output (#3347)
  > Document how to build applications against AWS-LC (#3386)
  > Add brainpoolP224r1, brainpoolP256r1, brainpoolP320r1, brainpoolP384r1, brainpoolP512r1 EC group support (#3286)
  > Avoid /usr/bin/env dependency in FIPS compiler wrapper (#3411)
  > BoringSSL: Fix beeu_mod_inverse_vartime on aarch64 (#3381)
  > Update MySQL CI integration to mysql-cluster-9.7.2 (#3409)
  > Fix `grpc-master-x86_64` integration test  (#3346)
  > ci: remove Graviton5 c9g fleet and job (#3400)
  > Share Bedrock model settings between autofix and util/backport (#3395)
  > Fix thread-local destructors with MSYS2 clang64 (#3405)
  > Prepare 5.5.0 (#3401)
  > Enable linker garbage collection for non-FIPS static builds (#3397)
  > Update concurrency in `security-review` (#3403)
  > Add MinGW shared-library Windows CI coverage (#3398)
  > Bump urllib3 from 2.6.3 to 2.7.0 in /tests/ci (#3372)
  > Update FIPS.md (#3382)
  > Fix docker image dependencies and update octocrab initialization (#3388)
  > Bump ring from 0.17.8 to 0.17.14 in /tests/ci/lambda (#3375)
  > Add Neoverse-V3 (Graviton5) detection and dispatch (#3374)
  > delocate: Drop redundant `-dI` flag (#3384)
  > ci: pin openssh required job (#3387)
  > Rename autofix reasoning role to AwsLcGitHubActionsBedrockRole (#3376)
  > Link header documentation from README (#3378)
  > Add tests for RSASSA-PSS with SHA-3 digest and MGF1 (#3377)
  > chore: simplify the api docs workflow (#3186)
  > Fix flaky s_client cipher tests by using better ciphers (#3373)
  > ci: harden security-review.yml against script injection (#3365)
  > Include <openssl/cipher.h> from <openssl/hmac.h> for OpenSSL compat (#3371)
  > Restore ADX/AVX2 code paths under OPENSSL_SMALL via new MY_ASSEMBLER_IS_TOO_OLD_FOR_ADX_AVX2 flag (#3368)
  > Bump to v5.4.0, add AES-KWP wrap/unwrap CASTs (#3366)
  > Use built-in Camellia in krb5 integration patch; fix CTS doc label (#3349)
  > Remove dead P-256 nistz asm under OPENSSL_SMALL and extend the size-check harness (#3350)
  > ci:ctest timing summary for MacOS (#3288)
  > Support for `EVP_AEAD_CTX_copy` (#3332)
  > Add AArch64 NEON runtime fallback for ML-KEM and ML-DSA (#3353)
  > Prepare 5.3.0 (#3354)
  > Add ACVP support for multi-expansion HKDF (#3352)
  > Bump golang.org/x/crypto from 0.47.0 to 0.52.0 in /util/vecgen (#3339)
  > Auto-fix integration patches and upload to S3 (#3324)
  > Bump log from 0.4.30 to 0.4.33 in /tests/ci/lambda in the cargo-ci-lambda group (#3329)
  > Prepare v5.2.0 (#3348)
  > Fix `SSL_OP_IGNORE_UNEXPECTED_EOF` being ignored over sockets (#3341)
  > Gate s2n-bignum _alt variants under OPENSSL_SMALL (#3320)
  > Add CTS mode, krb5 integration test (#3308)
  > Bump github.com/google/go-cmp from 0.6.0 to 0.7.0 in /tests/ci/x509/limbo-report in the gomod-limbo-report group (#3328)
  > Bump github.com/cloudflare/circl from 1.6.3 to 1.6.4 in /util/vecgen in the gomod-vecgen group (#3327)
  > openssl/target.h: Allow building for the e2k architecture (#3314)
  > ci: harden GitHub Actions workflows against script injection (#3323)
  > Pin required libssh2 integration to a release; track main separately (#3343)
  > Add ML-KEM decapsulate CASTs to break-kat tooling (#3342)
  > Update security review workflow (#3333)
  > Symbol versioning follow-ups: register peer cert APIs, fix dist_pkg_tests matrix (#3338)
  > Add Symbol Versioning Support (#3096)
  > Null-guard EVP_AEAD_CTX_cleanup for fork+shm safety (#3336)
  > Fix libgit2, xtrabackup, grpc tests and add OSSL3 peer cert APIs (#3331)
  > Documented and added feature stability tests for EVP_AEAD implementations that support concurrency through EVP_AEAD_CTX_seal/gather functions (#3325)
  > OPENSSL_SMALL: imply MY_ASSEMBLER_IS_TOO_OLD_FOR_512AVX on x86_64 (#3319)
Submodule mountpoint-s3-crt-sys/crt/s2n-tls f5f6c6c2c..bce022f41:
  > feat(metrics-subscriber): expose compatibility profile allow-lists (#6086)
  > fix: free client pub_key on all error paths in server shared secret (#6063)
  > fix: free encoded point on write failure in s2n_ecc_evp_write_params_point (#5964)
  > fix: zero-initialize s2n_tls13_keys in PSK binder functions (#6048)
  > fix: enforce JA4 list limit when building fingerprint (#6047)
  > fix: always free async offload op during connection teardown (#6080)
  > feat(s2n-tls-metrics): Add hello_retry_request_count (#6078)
  > fix(integration): replace BoringSSL fork git dependency with btls crate (#6067)
  > fix: bounds-check cert pkey type before array indexing (#6054)
  > feat(bindings): add signature_public_key_type to Connection (#6076)
  > fix(metrics): normalize s2n signature (#6077)
  > fix: restrict cert signature preferences in custom CNSA2 interop policies (#6073)
  > feat: skip blinding delay on missing required client cert (#6072)
  > feat(metrics): add negotiated metrics for compatibility (#6068)
  > feat: add `s2n_conn_get_signature_public_key_type` API (#5963)
  > chore(bindings): release bump v0.3.43 (#6070)
  > perf(sidetrail): Avoid digest zeroing in proof (#6056)
  > chore: upgrade CBMC to 6.11.0 (#6061)
  > fix: set actual_protocol_version_established on deserialized connections (#6049)
  > fix(bindings): avoid mutable aliasing in shared Config callbacks (#6053)
  > docs: io callback examples (#6036)
  > feat: add backwards compatible hybrid PQ policies (#6044)
  > chore: bump metrics subscriber for release (#6050)
  > fix(metrics): jitter export timing (#6046)
  > fix(cbc): Disable padding after key init and check decrypt len (#6042)
  > test: print assertion backtraces in FAIL_MSG_PRINT (#5785)
  > build(deps): bump cross-platform-actions/action from 1.3.0 to 1.4.0 in /.github/workflows in the all-gha-updates group (#6035)
  > fix(connection): Make set_config validator replacement transactional (#5847)
  > feat: add s2n_connection_handshake_complete() public API (#5906)
  > fix: Add fail-closed default cases to all switch statements (#6006)
  > fix: Zero-initialize all local variable declarations (#6007)
  > feat(metrics): flush on drop (#6032)
  > fix: fail explicitly when server selects unavailable KEM group in key share recv (#6026)
  > ci: Use GitHub mirror for musl and upgrade octokit action to Node 24 (#6031)
  > chore: update http test with new status code (#6034)
  > fix(bindings): default to efficient record size (#6020)
  > test: add behavior tests for Connection::wipe (#6029)
  > fix: handshake integrity over content types (#6017)
  > fix: zero-init hash state and guard async offload wipe against in-flight ops (#6028)
  > chore(bindings): release bump (#6027)
  > chore: remove inline policy builder code (#6025)
  > fix: prevent use-after-free in `Connection::set_config` when C setter rejects replacement (#6023)
  > fix(psk): return usage error for uninitialized offered PSK (#6009)
  > test: peer_cert_chain behavior with elided CA (#5914)
  > fix(docs): Hide unstable-renegotiate poll_send cfg in rustdoc (#6022)
  > fix: null `evp_cipher_ctx` after free in `s2n_session_key_alloc()` error path (#6024)
  > fix(ci): update msrv check to use Cargo.toml (#6021)
  > feat: add custom CNSA2 interop policies (#6011)
  > docs(build): clarify OpenSSL FIPS support and shared responsibility (#5856)
  > docs: document threading contract on s2n_async_pkey_op_apply and reje… (#5845)
  > fix: emit trailing colons for IPv6 addresses ending in a zero run (#5842)
  > fix: ensure record integrity over header version (#6014)
  > build(deps): bump actions/stale from 10 to 11 in /.github/workflows in the all-gha-updates group (#6016)
  > refactor: use header struct for decryption (#6004)
  > docs: add resources about Coordinated Vulnerability Disclosure to SECURITY.md (#5927)
  > build(deps): update crabgrind requirement from 0.2 to 0.3 in /tests/regression in the all-cargo-updates group across 1 directory (#5972)
  > style(integv2): run latest ruff format & check (#6005)
  > ci: update stale patch files for clang-format (#6000)
  > build(deps): bump the all-gha-updates group across 1 directory with 4 updates (#5994)
  > fix: use leaf public key NID for ML-DSA signature scheme matching (#5997)
  > feat: Include the error code in s2n_error Display impl (#5988)
  > fix: prefer clang for libs2n LTO to work under rust-lld (#5996)
  > chore: s2n-tls rust bindings v0.3.41 release (#5998)
  > fix(duvet): Correct CCS quote attribution to RFC 8446 section 5 (#5999)
  > Merge commit from fork
  > Merge commit from fork
  > fix: guard against underflow in record wipe length (#5973)
  > feat(docs): Render Cargo feature-gated code in docs (#5967)
  > ci: upgrade cppcheck from 2.3 to 2.13 via apt (#5943)
  > ci: add MSRV consumer check (#5986)
  > feat: add per-message handshake timing instrumentation (#5903)
  > chore: version bump (#5984)
  > feat: add client issue metrics (#5979)
  > refactor: export trait owns MetricRecord (#5981)
  > docs: Windows MinGW support is available (#5970)
  > chore: delete unused functions (#5978)
  > test: enable rust bindings tests on Windows (#5969)
  > ci: bump MSRV for extended workspace from 1.89 to 1.91 (#5980)
  > refactor(metrics): make record update infallible (#5976)
  > test: client_hello retrieval with HRR (#5975)
  > feat(bindings): add init feature flag (#5831)
  > build(deps): update openssl-src requirement from 300.5 to 400.0 in /bindings/rust/standard (#5971)
  > feat: enable rust bindings build and tests on Windows (#5958)
  > chore: metrics-subscriber release v0.0.4 (#5965)
  > refactor: two more self talk tests to use in-memory io pair (#5950)
  > refactor: fork based handshake tests to use in memory io pair (#5940)
  > chore: bindings release 0.3.39 (#5962)
```
</details>

### Does this change impact existing behavior?

No breaking changes. There are a few behaviour changes inherited from
the CRT worth calling out:
- `aws-c-s3` now retries `502 Bad Gateway` and `504 Gateway Timeout`
responses
([aws-c-s3#671](awslabs/aws-c-s3#671)).
- `aws-c-s3` now disables Nagle's algorithm on S3 connections
([aws-c-s3#659](awslabs/aws-c-s3#659)).

### Does this change need a changelog entry? Does it require a version
change?

Yes.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

Signed-off-by: Yerzhan Mazhkenov <20302932+yerzhan7@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jamesbornholt jamesbornholt jamesbornholt approved these changes
@dannycjones dannycjones dannycjones approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@monthonk @jamesbornholt @dannycjones