Backslash to Forward slash correction

[acrois]

It seems that browsing to a URL like this: https:// possibility.com/Clearing/Images\brnbagbk.gif (github encodes these URLs so I had to put a space after "https://") with a backward slash will resolve properly in browsers (latest Google Chrome and Firefox tested) by changing it to a forward slash when the URL is processed. It will automatically replace the backward slashes with forward slashes.

When it comes to storing the response in WARC as well as tools retrieving the correct URL (similar to how a browser would correct the request) I am not sure of the implications. I can imagine a process where a URL that resolves 404 with a backward slash in it can be retried with the backslashes replaced with forward slashes.

From the dashboard, when viewed in Google Chrome:

What is the best way to handle this in grab-site?

[TheTechRobo]

I might send a PR, but I don't really understand the codebase that well. 🤕

[ivan]

Browsers convert backslashes to slashes when they parse hrefs. They also do other odd things there, like trim leading and trailing whitespace.

I believe the right place to implement this would be in the href parsing in ludios_wpull.