tabby icon indicating copy to clipboard operation
tabby copied to clipboard

[Context/Developer docs] Exclude some URL query parameters

Open comxd opened this issue 1 year ago • 0 comments

Being able to define a list of query parameters for the crawled URL would make it possible to avoid, for example, crawling documentation in several languages.

In the following logs for a crawl job, I would have liked to exclude the hl parameter:

Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=it
[INFO]: Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=ja
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=pt-br
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=es-419
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=de
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=fr
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=zh-cn
...

Please reply with a 👍 if you want this feature.

comxd avatar Sep 18 '24 14:09 comxd