backstop-crawl icon indicating copy to clipboard operation
backstop-crawl copied to clipboard

Published 20 hours ago • verified publisher icon fffunction

Ignore certain urls by string

Open qiigodev opened this issue 6 years ago • 1 comments

It would be nice to be able to ignore urls that contain certain strings in an array.

Example: --ignore-url-containing = ["/blog,/tag,/news"]

qiigodev avatar Jan 25 '19 16:01 qiigodev

Great idea! I think best would be to also allow wildcard matching like:

--ignore-urls=/abc/*

what would be a typical separator?

JPustkuchen avatar Jan 20 '22 16:01 JPustkuchen