Is StripQueryString being on by default a good idea?

[osterman]

trafficstars

what

• Our documentation references the following URL https://marketplace.visualstudio.com/items?itemName=EditorConfigTeam.EditorConfig
• htmltest insists it is a 404, but it's 100% a 200
• Using htmltest 0.9.1

why

• multiple attempts to reproduce from home and office all return 404 in htmltest
• curl is happy
• curl forging htmltest User-Agent is also happy

Forge htmltest UA (200 OK)

curl -v -H'User-Agent: htmltest/0.9.1' https://marketplace.visualstudio.com/items?itemName=EditorConfigTeam.EditorConfig > /dev/null

Standard Curl UA (200 OK)

curl -v https://marketplace.visualstudio.com/items?itemName=EditorConfigTeam.EditorConfig > /dev/null

Chrome UA (200 OK)

curl -v -H'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36' https://marketplace.visualstudio.com/items?itemName=EditorConfigTeam.EditorConfig

[osterman]

Also, --log-level 0 does not return any useful information. I suggest debug mode (0) return the raw request/response from the origin to make debugging easier.

[wjdp]

Hi @osterman. My first thought was that we also send the header Range: bytes=0-0 (cuts down on downloads) and the server of visualstudio.com may not be handling this correctly but this does not seem to be the case. The following also works:

curl -v -H'User-Agent: htmltest/0.9.1' -H'Range: bytes=0-0' "https://marketplace.visualstudio.com/items?itemName=EditorConfigTeam.EditorConfig" > /dev/null

I'm out of ideas, as you mention it would be very helpful for debug mode to give more detail on the request/response. I will have a look at that when I get some time. For the moment I suggest you put affected URLs in IgnoreURLs.

[osterman]

Thanks @wjdp ! I've ignored the URL for now. It's a pretty low priority =)

[wjdp]

Hi @osterman, just had another look into this. The reason for this is sadly not a go bug but a 'feature' in htmltest :man_facepalming: We strip the query string off URLs before checking it which in this case ends up in a 404 (https://marketplace.visualstudio.com/items).

You can fix this by:

• Setting StripQueryString to false to disable this feature entirely
• Adding the URL to the exclude list StripQueryExcludes for this feature.

See the config section of the README for more details.

I will leave this issue open to review whether this feature should be turned on by default. I'm pretty sure it was a good idea at the time, but I can't quite remember what the reasoning was.

Thanks for your patience!