crawly
crawly copied to clipboard
Suggestion for nameing of closespider_timeout in config
It seems that when all the workers’ requests list are empty, the crawler still cannot stop automatically.
Maybe it is more convenient to determine when the spider need be stopped by providing a optional item closespider_deltacount
with default value 0
in config
This issue can be solved by setting closespider_timeout: 0
But I feel that the name of this settings item is confusing.
It is easy to misunderstand that this is a setting about the timeout of network requests
Maybe closespider_min_delta
is better, this is just my personal suggestion.
I see, yes, setting naming is an issue, but that's what the docs are for :sweat_smile: Interesting that the spider doesn't auto close when there isn't any requests in the queue, might be an issue with default config.