aws s3 sync --delete: delete files _after_ uploading

[tusbar]

Hello,

With awscli==1.7.37, when using s3 sync --delete, is there a way to have the files be deleted after the rest is uploaded? It seems like they’re always deleted before uploading.

$ aws s3 sync --delete dist s3://bucket
delete: s3://bucket/scripts/bundle.65406bc3e1133110.js
upload: dist/index.html to s3://bucket/index.html
upload: dist/scripts/bundle.9ad9b21077147aa5.js to s3://bucket/scripts/bundle.9ad9b21077147aa5.js
upload: dist/styles/bundle.6de17660ab595a44.css to s3://bucket/styles/bundle.6de17660ab595a44.css

[mtdowling]

What's the reasoning for waiting to delete until after all uploads are complete? If we did that, then we'd have to keep every delete request in memory, which could consume a large amount of memory for large buckets.

[tusbar]

@mtdowling all the reasoning that’s behind rsync’s --delete-after.

For example, if the transfer fails while syncing, let’s say right after the delete, I’ve deleted the old files, but haven’t added any of the new, which could be a problem.

[mtdowling]

Interesting. I suppose we could also evaluate adding other delete strategies similar to the delete options offered by rsync. I'll mark this as a feature request, and we'll add it to our backlog.

[tusbar]

A quick workaround is to run the sync command twice, once without --delete, then with the --delete flag.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[mehditlili]

Any updates on this? the feature delete_after is important IMHO in order to clean up source systems that have been synced

[tusbar]

@mehditlili use rclone instead.

[balloob]

Run your command twice, second time with --delete will delete the files after and won't re-upload the files the second time you run it.

[lcswillems]

Run your command twice, second time with --delete will delete the files after and won't re-upload the files the second time you run it.

That's a nice workaround! However it is slower than if the command was run once with the deletes at the end.

[unfor19]

@mehditlili use rclone instead.

Thank you for sharing this holy grail! This is exactly what I needed - rclone delete-after.

For deploying a static website to S3, I can safely say that storing the website's content in memory due to delete-after, which usually takes up to 30MB (depending on the website), is ok.

Thanks again!