scrapy-proxies icon indicating copy to clipboard operation
scrapy-proxies copied to clipboard

Proxy is set only if proxy_user_pass exists

Open DKanarsky opened this issue 3 years ago • 1 comments

I looked through your code and found out that proxy meta value was set only when there was proxy_user_pass from proxy record:

def process_request(self, request, spider):
	# Don't overwrite with a random one (server-side state for IP)
	if 'proxy' in request.meta:
		if request.meta["exception"] is False:
			return
	request.meta["exception"] = False
	if len(self.proxies) == 0:
		raise ValueError('All proxies are unusable, cannot proceed')

	if self.mode == ProxyMode.RANDOMIZE_PROXY_EVERY_REQUESTS:
		proxy_address = random.choice(list(self.proxies.keys()))
	else:
		proxy_address = self.chosen_proxy

	proxy_user_pass = self.proxies[proxy_address]

	if proxy_user_pass:
		request.meta['proxy'] = proxy_address
		basic_auth = 'Basic ' + base64.b64encode(proxy_user_pass.encode()).decode()
		request.headers['Proxy-Authorization'] = basic_auth
	else:
		log.debug('Proxy user pass not found')
	log.debug('Using proxy <%s>, %d proxies left' % (
			proxy_address, len(self.proxies)))

Have I missed smth?

DKanarsky avatar Jul 18 '22 20:07 DKanarsky

Agreed...it's missing the request.meta['proxy'] = proxy_address in the else statement.

0xspeedrunner avatar Aug 28 '22 02:08 0xspeedrunner