fscrawler icon indicating copy to clipboard operation
fscrawler copied to clipboard

Published 20 hours ago • verified publisher icon dadoonet

Add a Web Crawler

Open dadoonet opened this issue 5 years ago • 8 comments

We can base our code on https://github.com/yasserg/crawler4j

dadoonet avatar Feb 22 '19 16:02 dadoonet

@dadoonet, Thank you very much for the plugin. we are using it for crawling local files.

It would be great to include a Web crawler. Is there any timeline to include a web crawler?

rajasekhar-gundala avatar Jul 24 '22 04:07 rajasekhar-gundala

I honestly don't know if I will support this as Elastic now has this feature available with the basic license.

dadoonet avatar Jul 24 '22 07:07 dadoonet

I honestly don't know if I will support this as Elastic now has this feature available with the basic license.

@dadoonet, Yes Elastic has Web Crawler for Site Search. It would be good to have it for Workplace Search. (At least using the crawled documents in Workplace Search). Also, I don't think we can crawl authenticated sites using it.

rajasekhar-gundala avatar Jul 26 '22 04:07 rajasekhar-gundala