Options to no-check-md5 and skip-existing are ignored

[chrisspen]

I'm trying to sync two S3 buckets with the command:

chimit s3cmd sync --delete-removed --no-check-md5 --skip-existing s3://mysourcebucket s3://mydestbucket

and every time I run it, it reports:

Summary: 1598 source files to copy, 0 files at destination to delete

Even if I let it complete, and then re-run it, it reports that it again has to transfer 1598 files all over again, indicating that it's ignoring either the --no-check-md5 option or --skip-existing option.

The expected behavior, especially if --skip-existing is specified, is for s3cmd to completely ignore the files its already transferred. To not do this for buckets with a large number of files means that s3cmd wastes a huge amount of time re-transferring files unnecessarily. If the transfer completes, and I re-run it, with no changes having been made to any of the files, I would expect s3cmd to report "0 source files to copy".

[jchook]

I am also having this issue.

s3cmd sync --requester-pays --skip-existing --no-check-md5 s3://mybucket/myprefix .

Steps to reproduce:

1. Start sync with the command above
2. Interrupt sync after transferring about 50/100 files
3. Run the exact same command

Expected behavior

I expected s3cmd to ignore existing files, and continue syncing at file 51/100.

Actual behavior

s3cmd does not ignore existing files in the download folder. It starts from the very first file, re-downloading and and overwriting existing files.

Environment

• s3cmd version 2.0.1 installed via pip
• Ubuntu 16.04 LTS
• zsh

Notes

• The issue was happening on 1.6.1 (installed via aptitude). I upgraded to 2.0.1 in hopes that it would fix, but it did not.
• The data is being downloaded to a local directory located on a 2nd internal data-only ext4 drive

[fviard]

Thank you for your reports both of you, that helped me narrow down the 2 cases that produce such an issue.

• sync root bucket to bucket in the case of @chrisspen
• sync a source with a prefix in the case of @jchook that doesn't match with a single file, but potentially with multiple files, with a local destination.

I will look for a fix for these issues.

[jimmywan]

Any updates on this? Unfortunately s4cmd doesn't appear to support this either. :(

[matthewboman]

Still getting this issue with --skip-existing between two buckets

[kakadais]

This only happens between two buckets?

In my case, sync from local storage to bucket, --skip-existing looks like default option. Which means, existing files are skiped automatically, so now I'm searching for opposit option to 'overwirte existing files'.

So, any clear explanations?

• Intel Mac 11.1 / s3cmd 2.1.0

[jakubgs]

Also seeing this issue. Specifying --skip-existing and --no-check-md5 has no effect:

INFO: Running stat() and reading/calculating MD5 values on X files, this may take some time...