Properly use ProxyBroker with Selenium

[audi99]

What is the correct way to scrape N amount of proxies and then use them in a loop? for instance, here is my code:

import asyncio
from proxybroker import Broker
from selenium import webdriver


async def show(proxies):
    proxy_list = []
    while True:
        proxy = await proxies.get()
        if proxy is None:
            break
        # check accurately if the proxy is working
        if proxy.is_working:
            protocol = 'https' if 'HTTPS' in proxy.types else 'http'
            line = '{0}://{1}:{2}'.format(protocol, proxy.host, proxy.port)
            proxy_list.append(line)

    return proxy_list


def proxy_get():
    proxies = asyncio.Queue()
    broker = Broker(proxies)
    tasks = asyncio.gather(broker.find(types=['HTTP', 'HTTPS'], countries=['US', 'GB', 'CA'], limit=10), show(proxies))
    loop = asyncio.get_event_loop()
    results = loop.run_until_complete(tasks)
    return results


if __name__ == '__main__':
    for n in range(0, 10):
        print('run #{0}'.format(str(n)))
        proxies = proxy_get()[1]
        count = 1
        for proxy in proxies:
            print("Loop #{0}".format(str(count)))
            count += 1
            print("Using proxy: {0}".format(str(proxy)))
            chrome_options = webdriver.ChromeOptions()
            chrome_options.add_argument('--proxy-server=%s' % proxy)
            chrome_options.add_argument('headless')
            chrome = webdriver.Chrome(r'C:\chromedriver.exe', options=chrome_options)
            chrome.get('http://httpbin.org/ip')
            output = chrome.find_element_by_css_selector("body > pre").text
            print("Site output: {0}".format(output))
            print()
            chrome.quit()

on the 4th loop consistently, it seems to fail and give me a RuntimeError: Could not get external IP

Apologies if this is a moronic/obvious question, but it has had me stumped for quite awhile.

[synchronizing]

Instead of gathering proxies from ProxyBroker perhaps it would be more beneficial to use ProxyBroker's serve functionality, which will just serve you a single IP that you can use for anything. That single IP, which is ProxyBrokers, will automatically switch between proxies when there is some sort of error on the response request. 👍

[timot00]

Instead of gathering proxies from ProxyBroker perhaps it would be more beneficial to use ProxyBroker's serve functionality, which will just serve you a single IP that you can use for anything. That single IP, which is ProxyBrokers, will automatically switch between proxies when there is some sort of error on the response request. 👍

hello:) Thank you for your answser! I would be very interested to know how to do that robustly.. open a selenium firefox browser with 1 IP, get to a page, close browser, open immediately a new browser with another ip...

[synchronizing]

Instead of gathering proxies from ProxyBroker perhaps it would be more beneficial to use ProxyBroker's serve functionality, which will just serve you a single IP that you can use for anything. That single IP, which is ProxyBrokers, will automatically switch between proxies when there is some sort of error on the response request. 👍

hello:) Thank you for your answser! I would be very interested to know how to do that robustly.. open a selenium firefox browser with 1 IP, get to a page, close browser, open immediately a new browser with another ip...

So... that's not quite what the serve functionality will do. Instead, it will utilize a single IP until that IP is unable to reach the website you are shooting requests at. At that point it is when ProxyBroker switches proxy.

[timot00]

oh ok!!! thanks I didn't get it. so I'm gonna look at the common use of proxybroker.. thank you have a good day

Le lun. 23 déc. 2019 à 03:17, Felipe Faria [email protected] a écrit :

Instead of gathering proxies from ProxyBroker perhaps it would be more beneficial to use ProxyBroker's serve functionality, which will just serve you a single IP that you can use for anything. That single IP, which is ProxyBrokers, will automatically switch between proxies when there is some sort of error on the response request. 👍

hello:) Thank you for your answser! I would be very interested to know how to do that robustly.. open a selenium firefox browser with 1 IP, get to a page, close browser, open immediately a new browser with another ip...

So... that's not quite what the serve functionality will do. Instead, it will utilize a single IP until that IP is unable to reach the website you are shooting requests at. At that point it is when ProxyBroker switches proxy.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/constverum/ProxyBroker/issues/94?email_source=notifications&email_token=AKST3BF4U6UDBZ5BCIH5FODQ2ANUJA5CNFSM4GAJZKQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHQAR2Y#issuecomment-568330475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKST3BH3ON4ZIPHX2ZUUF2LQ2ANUJANCNFSM4GAJZKQQ .