w0d4
[Request] Download multiple videos from queue at once
It would speed things up greatly if yt-backup could spawn X number of youtube-dl commands to download multiple videos at once.
Perhaps in the config.json it could be specified, otherwise I think 3 would be a good default.
Some info that may help with this:
2nd reply, using multiprocessing: https://stackoverflow.com/questions/50197643/youtbe-dl-multiple-downloads-at-the-same-time.
Let me know your thoughts 😃
In general a good idea I think. But, I expect to get massive problems with youtube throwing HTTP 429 errors. The HTTP 429 problems were my main reasons to write this. To implement multithreaded downloading, I have to rewrite the whole download logic, since aftere each run of youtube-dl, everything from download_dir will be moved away via rclone. Not a good idea if there are 10 other youtube-dl processes which are currently writing files to the same directory :smile: Also I expect massive database locking problems, since the current database logic was not written with multhreading in mind. Maybe I will start testing this one day in an extra branch, seperate from stable master.
Ah I see, sounds like a big task. I've personally never been hit with the 429 error, possibly because my youtube-dl download speed is so slow (<2MB/s).
Shall I close this issue for the time being?
No, the issue is fine. I tagged it as improvement. So it's visible, it's not a bug but a feature request.