s3-sync-action
s3-sync-action copied to clipboard
jakejarvis
Upload only changed files
Hello, I don't really have an issue, its more like a request, would it be possible to make it only upload the changed files ? I have a big repository and its reuploading every file.
If its not possible that's okay too, this is already very helpful, good job!
I have the same issue with this action, but not when I sync folders locally. I wonder if it might have something to do with the temporary environment that GitHub Actions run in, and it causing aws to incorrectly identify all files as modified…?
Found a working solution from another thread. Add '--size-only' without quotes to your args parameter. https://github.com/aws/aws-cli/issues/5216
@chrissingh Yes! That worked for us. You just helped me cut the running time of our sync-job from 35 minutes (nominal, 90 minutes worst case) to 3.5 minutes. Where do I send the beer?
I have this issue and also that people change files but have same byte count frequently enough to be an issue.
Unfortunately, there has to be custom code because this use the cli aws s3 sync behind the scenes (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) and what we really need is a hash comparison.
Details here: https://stackoverflow.com/questions/75555482/how-to-use-aws-s3-sync-to-sync-modified-files-that-still-have-the-same-size
If you are using this action when merging a branch to main, one idea is to call git diff --raw HEAD^ HEAD to get a list of the affected files in the merge, then construct an --include statement for the s3 sync based on that.
A PR would be out-of-scope for this project though, which is really meant for the broader use case of syncing directories to S3 whenever the action is run. But it would be a nice fork IMHO. I don't see any kind of "S3 Sync upon Merge" action in the Marketplace.
