tabby
tabby copied to clipboard
[Context/Developer docs] Exclude some URL query parameters
Being able to define a list of query parameters for the crawled URL would make it possible to avoid, for example, crawling documentation in several languages.
In the following logs for a crawl job, I would have liked to exclude the hl parameter:
Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=it
[INFO]: Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=ja
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=pt-br
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=es-419
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=de
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=fr
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=zh-cn
...
Please reply with a 👍 if you want this feature.