Repository navigation
Is there parity for aws s3 sync? #824
Description
Activity
The
aws-sdkgem does not provide this functionality. As far as I know, none of the AWS SDKs, except the CLI have a filesystem to S3 sync function.Reacted by Joel Courtney, mlevyLI, shelby spees, David Pell, Alexey Pikin, Trevor Copeland, Jo Boiteau, Shaunak Vairagare, Angelo Romualdo, Pavel Dotsulenko and 21 more:(
Reacted by Joel Courtney, Tim Sabat, Tim, Mariusz Kozieł, einSelbst, David Revelo, mlevyLI, shelby spees, David Pell, Alexey Pikin and 20 moreAs far as I know, none of the AWS SDKs, except the CLI have a filesystem to S3 sync function.
Hi, just wanted to stop by and add my :( to this. Sync functionality has been available in the PHP SDK since 2013. The Java SDK also has this/something similar.
Reacted by Shaunak Vairagare, Pavel Dotsulenko, Ryutaro Yamada, Sampson Crowley, DrewCurd, Sean Dunford, Richard Baptist, Tata, David Revelo and Matt HampelSame here.. just stopping by again to add my :(
This is a really useful feature to have in sdk, so we don't have to install cli dependency on hosting, and our background jobs can run purely on gem dependency.
Adding this feature will make it whole lot easier to do PaaS deployments on system like Heroku.
Reacted by Sampson Crowley, mrbean, Erika Miguel, Preshoth Paramalingam, Sean Dunford, Richard Baptist, Adam Gaskins, Harlan Barnes, David Pell and JatinReacted by Angelo Romualdo, Pavel Dotsulenko, Neo Sepulveda, Chris Wallace, Richard Tong, Hani Alsioufi, mrbean, Erika Miguel, Preshoth Paramalingam, Sean Dunford and 3 moreBumping for 2019. Any chance of having this function added to the Ruby SDK?
Reacted by Sean Dunford, Brian Warsing and Shaunak VairagareLooking for SDK equivalent SDK method for
aws s3 sync [SOURCE] s3://[BUCKET]/[KEYPREFIX] --exact-timestampsDoes this exist?
Reopening as a feature request. Happy to take an PRs.
Reacted by dpikt, Adam Gaskins, Sathish N, SudoDevNull, Adam Dry, Shaunak Vairagare and Jay PrallReacted by George Viaud, Shaunak Vairagare and David ReveloNeed the ability to run:
aws s3 sync s3://accountOne/sourceBucket/etc s3://accountTwo/targetBucket/etc --delete --include="*"Using the API, you can create a TransferManager object, but it assumes a single S3 AWS connection. An API change would hopefully accommodate this use case.
Any update on this?
We are in 2021, any updates?
Reacted by Aymen Bou, James Robson, Will Munn, George Smith, Shaunak Vairagare and Thorsten Böttger^ @VitalyTarasiu I can second this. We are in 2021.
Still waiting for this feature to drop :(
+1
@pviet-cohere it's not "just a loop". May be read up on the documentation for the original utility?
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
Reacted by Adam Dry, Jatin and David ReveloHi guys, I also am looking for SDK equivalent of CLI aws sync; I found this really helpful article that served my purpose. The code is easy to run as well.
https://blog.bitsrc.io/a-practical-guide-to-building-a-node-js-service-on-an-aws-s3-bucket-aff19105ba83Nevertheless, I do request aws-sdk to include the sync feature to standardize the algo.
I know my comment will be unpopular; I braced for
☹️ .
But, what's wrong with installing the AWS CLI? If you don't want it on the host, you can also install it inside a container.I personally will choose neither SDK nor CLI for sync. Because, IMO, there is an economy behind software.
Old programmers are starting to die, literally; software, although still young on average, can live much longer than humans.
We are heading in the direction where there are not enough humans to maintain software; we are outnumbered.
If not enough people are using it, it is potentially not viable.You'll get better priority, support, and maintenance by switching to something more specific, like Rclone.
Andposix-spawnisn't too much of a pain, is it?Anyway, it seems the AWS team is still open for the PR; if you really want this, you should go for it.
Hi and apologies for the very long silence here. Good news: the Ruby SDK now has a high-level
TransferManagerthat can recursively transfer a whole directory tree to/from S3, which meets the spirit of what you were after:require 'aws-sdk-s3' tm = Aws::S3::TransferManager.new tm.upload_directory('/path/to/local/dir', bucket: 'my-bucket', recursive: true) tm.download_directory('/path/to/local/dir', bucket: 'my-bucket')
More detail and examples here: Transfer Manager directory support for the AWS SDK for Ruby.
One honest caveat: this does a full recursive transfer rather than a true
aws s3 sync, so it doesn't yet compare size/modified-time to skip unchanged files. Since the directory-transfer support covers the core of this request, I'm going to close this out. If you'd specifically like the delta/copy-only-changed behavior, please open a new issue for i, and of course feel free to open anything else or ask questions.Reacted by Joel Courtney and davidReacted by Joel Courtney and davidReacted by davidThis issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
The AWS CLI has the the amazing
aws s3 synccommand which I use frequently. It's super fast because in some cases it only copies the delta between the buckets.There doesn't seem to be a similar way to invoke that API from this library. Is it here and I'm just missing it?