cget
cget copied to clipboard
Replace CGetURLOpener with urlretrieve
For reasons I have not entirely traced, URLopener (and thus FancyURLopener) does not work with some proxies. I can see that instead of setting up a tunnel with a CONNECT request, it makes a GET request to the proxy, which in my case induces a 502-bad-gateway response. Even though it recognises the proxy settings, it doesn't quite do the right thing.
However, urlopen uses ProxyHandler to accommodate the situation, and does work through this troublesome proxy. urllib.request.urlretrieve has the same interface as URLopener.retrieve, and seems like a drop-in replacement. I added a try/except around the urlretrieve to get the BuildError for responses of 400 or greater, since there is no longer a custom http_error_default or a good place to hang it.
This change addresses my internal build problem and probably also issues #52 and #77. Plus, no more deprecation warnings.
Hi @pfultz2 , may I ask why this is not merged yet? It certainly sorted the issues I was facing with an internal proxy :)!
There is failures in ubuntu CI, if you can merge the latest from master which updated it to 20.04.