frontera
frontera copied to clipboard
Queued remain as queued when you stop crawling in sqlalchemy backend.
Ideally, Queued items should be put back as not crawled when the spider is closed.
Good finding again! I'm not sure that will help to completely avoid queued status, when it's not actually queued. Spider process can be killed, so everything in queue will be lost. But in general, I agree, we shouldn't loose cached urls in the queue.