scrapy-proxies
scrapy-proxies copied to clipboard
Proxy is set only if proxy_user_pass exists
I looked through your code and found out that proxy meta value was set only when there was proxy_user_pass from proxy record:
def process_request(self, request, spider):
# Don't overwrite with a random one (server-side state for IP)
if 'proxy' in request.meta:
if request.meta["exception"] is False:
return
request.meta["exception"] = False
if len(self.proxies) == 0:
raise ValueError('All proxies are unusable, cannot proceed')
if self.mode == ProxyMode.RANDOMIZE_PROXY_EVERY_REQUESTS:
proxy_address = random.choice(list(self.proxies.keys()))
else:
proxy_address = self.chosen_proxy
proxy_user_pass = self.proxies[proxy_address]
if proxy_user_pass:
request.meta['proxy'] = proxy_address
basic_auth = 'Basic ' + base64.b64encode(proxy_user_pass.encode()).decode()
request.headers['Proxy-Authorization'] = basic_auth
else:
log.debug('Proxy user pass not found')
log.debug('Using proxy <%s>, %d proxies left' % (
proxy_address, len(self.proxies)))
Have I missed smth?
Agreed...it's missing the request.meta['proxy'] = proxy_address in the else statement.