frontera icon indicating copy to clipboard operation
frontera copied to clipboard

Queued remain as queued when you stop crawling in sqlalchemy backend.

Open RajatGoyal opened this issue 10 years ago • 1 comments

Ideally, Queued items should be put back as not crawled when the spider is closed.

RajatGoyal avatar Sep 11 '15 14:09 RajatGoyal

Good finding again! I'm not sure that will help to completely avoid queued status, when it's not actually queued. Spider process can be killed, so everything in queue will be lost. But in general, I agree, we shouldn't loose cached urls in the queue.

sibiryakov avatar Sep 17 '15 03:09 sibiryakov