scrapy-rotating-proxies
scrapy-rotating-proxies copied to clipboard
Proxy-Authorization added
I think it's an important feature because not always people have an unauthorized proxy.
We had the following error before:
scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy <proxy> [{'status': 407, 'reason': b'Proxy Authentication Required'}]
Please, update PyPI after merging this request.
Thanks, confirm it is working!
Thanks for this, can confirm its a big issue, this is forcing me to use forks rather than the main package
@StasDeep, is it possible to merge and close this PR. Seems like it's an actual problem for some engineers.
@dtalkachou I don't have merge permissions on this repository. @kmike ?
Unfortunately I lost permissions myself :( We should probably fork this repo to another place, likely scrapy-plugins.
@kmike, Is there anybody else who has permissions?
Proxy-Authorization
header is set by Scrapy built-in HttpProxyMiddleware. @dtalkachou do you have it enabled?
Seems like I didn't have it enabled.
Yes, after code reading I think that HTTPPROXY_ENABLED = True
should solve the problem. If somebody checks this approach, let me know. And provide more details for developers who will have the same problem.