tumblr-crawler icon indicating copy to clipboard operation
tumblr-crawler copied to clipboard

[Bug] Unable to continue after failure...

Open momoe opened this issue 5 years ago • 3 comments

"Failed to retrieve video from <URL>" stays stuck here and cannot continue onto the next list, script has to be killed and restarting downloads from the beginning all over again.

momoe avatar Jul 17 '18 02:07 momoe

@momoe Actually it will try RETRY = 5 times. It won't get stuck.

# Setting timeout
TIMEOUT = 10

dixudx avatar Oct 08 '18 13:10 dixudx

As you see in above tickets it get's stuck despite RETRY and TIMEOUT parameters.

As interim solution I run it in a loop , each time with one site in sites.txt , and have watchdog on a side, which is a script that checks if amount of files in directory and subdirectories has changed, if not, than it kills it, here is watchdog: https://gist.github.com/gwpl/22b3c3e66711489b8d36cf8e5923c308

gwpl avatar Dec 10 '18 08:12 gwpl

#107 helps fixed this.

dixudx avatar Dec 10 '18 12:12 dixudx