scrapy-rotating-proxies icon indicating copy to clipboard operation
scrapy-rotating-proxies copied to clipboard

What about getting proxy list from url?

Open aidiss opened this issue 4 years ago • 2 comments

It could work as something like this

ROTATING_PROXY_LIST_URL = 'http://..../get_proxies.json'

Or, what would be the correct way of implementing this?

aidiss avatar Oct 05 '19 18:10 aidiss

  def getProxyList():
          resp = requests.get('https:/name/api/proxylist.txt')
          return resp.text.split()

  ROTATING_PROXY_LIST = getProxyList()

Same with json, you need to parse response of request as json

heilwood avatar Oct 26 '19 10:10 heilwood

How to update proxy list every month, without stop the scrapper app ??

hamcher avatar Apr 12 '23 12:04 hamcher