algolia-webcrawler icon indicating copy to clipboard operation
algolia-webcrawler copied to clipboard

Manage url that have non encoded characters

Open beaubbe opened this issue 3 years ago • 0 comments

If the crawler tries to request a url that has non encoded characters, it will simply stop there and giv an error.

To fix this, url is first decoded to decoded what is already present in the url so that there wil not be a double encoding. It is then encoded.

beaubbe avatar Jul 28 '21 17:07 beaubbe