tumblr-crawler
tumblr-crawler copied to clipboard
[Bug] Unable to continue after failure...
"Failed to retrieve video from <URL>" stays stuck here and cannot continue onto the next list, script has to be killed and restarting downloads from the beginning all over again.
@momoe Actually it will try RETRY = 5
times. It won't get stuck.
# Setting timeout
TIMEOUT = 10
As you see in above tickets it get's stuck despite RETRY and TIMEOUT parameters.
As interim solution I run it in a loop , each time with one site in sites.txt , and have watchdog on a side, which is a script that checks if amount of files in directory and subdirectories has changed, if not, than it kills it, here is watchdog: https://gist.github.com/gwpl/22b3c3e66711489b8d36cf8e5923c308
#107 helps fixed this.