Support ignoring URL params #90

[bluemindset]

I have implemented a new feature by adding a flag called ignore-query to the project. This flag strips query parameters from the URI during scraping. To achieve this, I utilized the NormalizeURL function in the utils.go file to ensure consistency in URL normalization.

I would love to hear your feedback on this implementation and contribute more to the project!

Best regards, Stefanos

[s0rg]

Thank you for your submission, but i think you did it wrong - not in terms of code, but in logic - this flag strips params only from starting url, but not from crawled ones so it will not work as expected.

best regards, Alexey

[bluemindset]

Hello, and thank you for the review. I believe I've now corrected the logic.