crawler icon indicating copy to clipboard operation
crawler copied to clipboard

Published 20 hours ago • verified publisher icon fredwu

Retrieving all items in Crawler.Store

Open mikhailbot opened this issue 6 years ago • 1 comments

It appears you can only find pages that have been crawled if you know the URL, there's no way to return all pages in the store. Is this by design, or just not a completed API for interfacing with the registry?

I ask because it's getting all the pages it seems but if I don't know the URL I've no access to get the information from the store?

mikhailbot avatar Jul 04 '19 22:07 mikhailbot

I'm also curious about that. Is there a possibility to get a list of items crawled?

christophpaster avatar Jan 02 '21 22:01 christophpaster

Added here: https://github.com/fredwu/crawler/commit/0ebc1132cc540c0538303bb7e70425a3380259fe :)

fredwu avatar Sep 28 '23 15:09 fredwu