s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

Published 20 hours ago • verified publisher icon peak

Sync command issue with many files.

Open eminegit opened this issue 2 years ago • 4 comments

Hi, the sync command works great with a few files. I am trying to sync 2 million files into the s3 bucket which has more than 15 million files but it stops after 5 minutes with a "killed" error message. I use AWS s3 sync command; it works fine but it is just slower than the s5cmd command.

eminegit avatar Jun 20 '22 04:06 eminegit

@eminegit Hello! Could you share your s5cmd version please? We have released a new version to address this issue (v2.0.0-beta.2)

sonmezonur avatar Jun 21 '22 06:06 sonmezonur

Hi, the version I have is v2.0.0-beta.2-c978e4a. I used 's5cmd_2.0.0-beta.2_Linux-64bit.tar.gz' file to install s5cmd.

eminegit avatar Jun 21 '22 13:06 eminegit

Sounds like this could be the same issue as in #441

misuto avatar Jul 06 '22 22:07 misuto

Sounds like this could be the same issue as in #441

I agree that this is related to it.

I've tried to reproduce error. I've tried to sync from a directory with huge number of files. I received fatal error: runtime: out of memory not a "killed" error in Linux. The errors probably means the same thing.

I've also outlined some of the possible solutions at https://github.com/peak/s5cmd/issues/441#issuecomment-1201413979.

kucukaslan avatar Aug 01 '22 16:08 kucukaslan