gpt-crawler icon indicating copy to clipboard operation
gpt-crawler copied to clipboard
verified publisher icon BuilderIO

Exclude directories

Open gthay opened this issue 2 years ago • 2 comments

Can someone add a function where you can also exclude specific directories? Like, don't crawl example.com/products/ (and all products deeper inside the path?

gthay avatar Nov 16 '23 09:11 gthay

sounds like a great option if you'd like to create a PR for it!

steve8708 avatar Nov 17 '23 00:11 steve8708

Opened a PR with exclude functionality. Simply leverages was crawlee already offers.

ciphercom avatar Dec 28 '23 20:12 ciphercom