rightmove_webscraper.py
rightmove_webscraper.py copied to clipboard
Adding proxies optional argument
I needed to use proxies for requests and added an optional argument to pass in when initializing RightmoveData.
If you don't want to use proxies - nothing changes, if you do, the init looks like this:
rm = RightmoveData(url, False, proxies)
where proxies are a dict:
proxies = { 'https' : 'https://user:password@proxyip:port' }
Apologies for the delay! I'm not clear on why this is necessary so hesitant to just implement it. Could you explain why/when it's needed? Would it be better to just allow any kwargs for request.get?
Also please clarify if it also needs to be added to this line:
status_code, content = self._request(weblink)