python-sitemap icon indicating copy to clipboard operation
python-sitemap copied to clipboard

Published 20 hours ago • verified publisher icon c4software

Stop and continue

Open ishandutta2007 opened this issue 5 years ago • 1 comments

The issue with this tool is once it halts, your have to start all over again from scratch. And with large sites this is a very common scenario. Since we already have the partially generated xml, it would be nice to continue from where it was interrupted. Let me know your thoughts on this and how to achieve this, I am willing to send pull request once I have a better understanding of the code

ishandutta2007 avatar Feb 22 '20 09:02 ishandutta2007

Hi,

It's a really nice idea. The major drawback I can see is that we can miss some new page in previously crawled pages.

But with some work (like preload all links previously crawled to avoid refetching) we can implement that kind of feature.

c4software avatar Feb 22 '20 09:02 c4software