wayback-machine-scraper icon indicating copy to clipboard operation
wayback-machine-scraper copied to clipboard

Published 20 hours ago • verified publisher icon sangaline

[Question] How to get latest crawl?

Open santoshbs opened this issue 3 years ago • 1 comments

Is there a way that I can get the most recent version (a single version) of a full site crawl of a list of URLs?

santoshbs avatar Dec 06 '20 23:12 santoshbs

This isn't supported by the current API, but adding an -l/--latest option seems like a good feature request. In the meantime, you might be interested in scrapy-wayback-machine. That project provides a Scrapy middleware that this project uses under the hood, and it offers a lot more flexibility in terms of customizing behavior.

I'll leave this open as a feature request.

sangaline avatar Feb 15 '21 16:02 sangaline