scrapy-rotating-proxies
scrapy-rotating-proxies copied to clipboard
What about getting proxy list from url?
It could work as something like this
ROTATING_PROXY_LIST_URL = 'http://..../get_proxies.json'
Or, what would be the correct way of implementing this?
def getProxyList():
resp = requests.get('https:/name/api/proxylist.txt')
return resp.text.split()
ROTATING_PROXY_LIST = getProxyList()
Same with json, you need to parse response of request as json
How to update proxy list every month, without stop the scrapper app ??