scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

Published 20 hours ago • verified publisher icon TeamHG-Memex

Proxy-Authorization added

Open dtalkachou opened this issue 3 years ago • 8 comments

I think it's an important feature because not always people have an unauthorized proxy.

We had the following error before:

scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy <proxy> [{'status': 407, 'reason': b'Proxy Authentication Required'}]

Please, update PyPI after merging this request.

dtalkachou avatar Mar 08 '21 16:03 dtalkachou

Thanks, confirm it is working!

dkarelov avatar May 30 '21 20:05 dkarelov

Thanks for this, can confirm its a big issue, this is forcing me to use forks rather than the main package

8W9aG avatar Jul 24 '21 01:07 8W9aG

@StasDeep, is it possible to merge and close this PR. Seems like it's an actual problem for some engineers.

dtalkachou avatar Feb 17 '22 22:02 dtalkachou

@dtalkachou I don't have merge permissions on this repository. @kmike ?

StanHENI avatar Feb 18 '22 05:02 StanHENI

Unfortunately I lost permissions myself :( We should probably fork this repo to another place, likely scrapy-plugins.

kmike avatar Feb 18 '22 13:02 kmike

@kmike, Is there anybody else who has permissions?

dtalkachou avatar Feb 18 '22 14:02 dtalkachou

Proxy-Authorization header is set by Scrapy built-in HttpProxyMiddleware. @dtalkachou do you have it enabled?

divtiply avatar Jun 07 '22 06:06 divtiply

Seems like I didn't have it enabled.

Yes, after code reading I think that HTTPPROXY_ENABLED = True should solve the problem. If somebody checks this approach, let me know. And provide more details for developers who will have the same problem.

dtalkachou avatar Jun 07 '22 08:06 dtalkachou