twitch-dl
twitch-dl copied to clipboard
ihabunek
--max-workers doesn't seem to help
Hey! I found that on my AWS instance (t3.large, 2 cores) downloading a large VOD (20gb) has better speed with -w 2 instead of -w 8 or any other value. On -w 2 it gives about 50MB/s, on -w 8+ it gives about 40MB/s, however my instance connection is about 250MB/s. Any ideas why could this happen?
$ curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python3 -
Retrieving speedtest.net configuration...
Testing from Amazon.com (...)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by HABNET (Frankfurt) [0.89 km]: 2.736 ms
Testing download speed................................................................................
Download: 2217.18 Mbit/s
Testing upload speed......................................................................................................
Upload: 2339.79 Mbit/s
$ twitch-dl download -q source -f mp4 --overwrite --keep -w 1 https://www.twitch.tv/videos/...
...
Downloaded VOD 105/1137 (9%) 821.4MB of ~8.7GB at 18.3MB/s remaining ~7 min 22 sec
$ twitch-dl download -q source -f mp4 --overwrite --keep -w 2 https://www.twitch.tv/videos/...
...
Downloaded VOD 138/1137 (12%) 1.1GB of ~8.7GB at 36.8MB/s remaining ~3 min 31 sec
$ twitch-dl download -q source -f mp4 --overwrite --keep -w 4 https://www.twitch.tv/videos/...
...
Downloaded VOD 344/1137 (30%) 2.6GB of ~8.7GB at 39.8MB/s remaining ~2 min 35 sec
$ twitch-dl download -q source -f mp4 --overwrite --keep -w 8 https://www.twitch.tv/videos/...
...
Downloaded VOD 634/1137 (55%) 4.8GB of ~8.6GB at 48.6MB/s
$ twitch-dl download -q source -f mp4 --overwrite --keep -w 32 https://www.twitch.tv/videos/...
...
Downloaded VOD 84/1137 (7%) 657.0MB of ~8.7GB at 46.9MB/s remaining ~2 min 55 se
Hm, this is an interesting issue.
When I was first developing twitch-dl, my individual VOD download speeds were quite low, nowhere near enough to saturate my download bandwith, so I cranked the default workers to 20 and that seemed to work well. Testing now, I don't see any improvement after 4-5 workers, or even less on lower bandwidth links.
Not sure why increasing worker count would decrease the download speed. This is something to test and try and work out...
Perhaps specific to certain CDNs? Obviously non-scientific, but from my 300mb/s home connection, it seems I can saturate my connection with 6 workers for VODs not already cached. For VODs already in the CDN cache, I usually can achieve 26MB/s with a single worker, which is really not bad for a single-thread HTTP download.
@garoto It seems you're right. I tried downloading a relatively old video from a less popular streamer which would not be cached. I downloaded it with 1 worker and got speed ~10MB/s. Then I stopped the download, deleted already downloaded vods and restarted the download with 1 worker and now I got ~20MB/s which is close to my link's capacity. So cache definitely plays a role.
Still, I'm wondering if I should decrease default worker count to something lower like 5.
5 seems like a more sane default to me. People can always increase the worker threads depending on how their nearest Cloudfront CDN behaves.