s5cmd
s5cmd copied to clipboard
'cp -n' performance
Copy command with "no clobber" performs worse than I expected on large batch copies. Few notes:
- From my testing a 'cp -n' that copies no files is actually slightly slower than a 'cp' command that transfers and overwrites all of the files.
- This is for copies of thousands of files from network drive to a high performance on premise s3 instance.
- Currently s5cmd loops through all source files and checks each individually against the destination to see if destination exists. This results in multiple s3 API calls per file.
- I would expect that grabbing a file listing from both src and dest, comparing, and then copy only the required files would perform much better in nearly all scenarios (I'm doing this for mine instead of using s5cmd and it's much faster)