crawly icon indicating copy to clipboard operation
crawly copied to clipboard

Suggestion for nameing of closespider_timeout in config

Open EdmondFrank opened this issue 3 years ago • 2 comments

image

It seems that when all the workers’ requests list are empty, the crawler still cannot stop automatically.

Maybe it is more convenient to determine when the spider need be stopped by providing a optional item closespider_deltacount with default value 0 in config

EdmondFrank avatar Apr 29 '21 02:04 EdmondFrank

This issue can be solved by setting closespider_timeout: 0

But I feel that the name of this settings item is confusing.

It is easy to misunderstand that this is a setting about the timeout of network requests

Maybe closespider_min_delta is better, this is just my personal suggestion.

EdmondFrank avatar Apr 29 '21 02:04 EdmondFrank

I see, yes, setting naming is an issue, but that's what the docs are for :sweat_smile: Interesting that the spider doesn't auto close when there isn't any requests in the queue, might be an issue with default config.

Ziinc avatar Sep 07 '21 18:09 Ziinc