s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

Published 20 hours ago • verified publisher icon peak

S5CMD numworker issue

Open anudina opened this issue 7 months ago • 1 comments

Hi I am trying to download lot of files from S3 to local. I am sending the command file like below s5cmd run --numworker 256 commands.txt and my commands file will have below sample of around 500 entries

eg: commands.txt

cp s3://XYZ/alprgagqenm-e-ec-c7kx-enmhost-1/abc.txt /home/test/copy/files/ I have literally like these around 500 entries in single commands.txt file the same thing will be repeated untill all the files from S3 are downloaded like 10000 iteration. What I am seeing is when I add records more than 70 its really taking a long to cp the files inspite of adding --numworker to 256 there is no impact.

  1. what am I doing wrong or how does num-worker works?
  2. Every number worker is 1 CPU on the server its running? for eg: i am running this on 80CPU 256GB gaint box. provide some insight to tweak this pls.

anudina avatar Jul 26 '24 15:07 anudina