tusd icon indicating copy to clipboard operation
tusd copied to clipboard
verified publisher icon tus

How to speed up transfer rate?

Open MarhabaLRS opened this issue 5 years ago • 14 comments

Why tusd is slow? When I start upload, Uploading chunk to my Tusd instance, after some time was halted upload, then resume the upload, then halted. It was repeatedly happened until complete the upload. Is there any reason?. It will take about 2times time to complete single upload

I have 60Mbps Connection. bandicam 2020-11-22 10-10-41-527

Setup details

  • Operating System: Ubuntu
  • Used tusd version: latest
  • Used tusd data storage: AWS S3
  • Used tusd configuration: -s3-bucket, -s3-endpoint, -s3-object-prefix files, -s3-part-size 33554432, -max-size 2199023255, -timeout 0, -port 1080, -behind-proxy true, -expose-metrics false
  • Used tus client library: tus-js-client

Colud you suggest setups for tus instance to achive maximum perfomance

MarhabaLRS avatar Nov 22 '20 05:11 MarhabaLRS

Sometimes it will be like this.

bandicam 2020-11-21 22-58-34-287

MarhabaLRS avatar Nov 22 '20 05:11 MarhabaLRS

How is your upload connection to AWS S3? tusd can only accept incoming data as fast as it can push it to AWS S3. So, if your upload speed to AWS S3 is not sufficient, tusd will also slow down.

Acconut avatar Nov 23 '20 09:11 Acconut

My PC(tus-js-client from localhost) to EC2 about 40-50 Mbps, EC2 to S3 shows 1Gbps.

MarhabaLRS avatar Nov 23 '20 12:11 MarhabaLRS

Ok, good to know. Can you try increasing the -s3-part-size flag to more than 50MiB (see https://github.com/tus/tusd/blob/master/cmd/tusd/cli/flags.go#L59)? Maybe try a few different values and see if they change the behavior.

Acconut avatar Nov 23 '20 13:11 Acconut

Current part size is 96MiB 100663260 Bytes . I've already tried 32MiB, 48MiB, 56MiB. These are the flags tusd -s3-bucket tusd-storage -s3-endpoint https://s3.eu-central-1.amazonaws.com -s3-object-prefix files -s3-part-size 100663260 -max-size 2199023255552 -timeout 0 -port 1080 -hooks-http http://0.0.0.0:8080/tus-auth.php -hooks-http-retry 3 -hooks-http-backoff 2 -hooks-enabled-events pre-create,post-create,post-finish,post-terminate,post-receive -behind-proxy true -expose-metrics false

MarhabaLRS avatar Nov 23 '20 13:11 MarhabaLRS

And do these different values result in different behaviour?

Acconut avatar Nov 23 '20 15:11 Acconut

Nop. Still the result is same. 1-3 Second Uploading in Maximum Speed, Then down for 1-3 Second. Sometimes upload get stuck. Tusd server not receiving data continuously.

MarhabaLRS avatar Nov 23 '20 17:11 MarhabaLRS

How about values larger than 96MiB?

Acconut avatar Nov 23 '20 18:11 Acconut

The result is same. I've tried 200MiB,500MiB and 1GB. How did you implemented this (https://tusd.tusdemo.net/) test server.

MarhabaLRS avatar Nov 24 '20 03:11 MarhabaLRS

The demo server is running the tusd Docker image on Heroku: https://github.com/tus/tusd/blob/master/.github/workflows/main.yml#L47-L60

Acconut avatar Nov 30 '20 22:11 Acconut

Any updates here?

arrowcircle avatar Nov 16 '22 14:11 arrowcircle

We have tried to investigate the root cause of slower uploads speeds to S3, but did not find a good solution so far.

I hope to share a more detailed report soon.

Acconut avatar Feb 11 '23 20:02 Acconut

Some updates and insights are available in https://github.com/tus/tusd/pull/924.

Acconut avatar Mar 26 '23 23:03 Acconut