Cannot setup authenticated proxy, that works in regular selenium

[tsoernes]

I have an authenticated proxy.

This works, with regular Selenium:

from selenium.webdriver import (Chrome, ChromeOptions)
proxy_url = f"https://{domain}:{port}"
opts = ChromeOptions()
opts.add_argument("proxy-server=" + proxy_url)
driver = Chrome(options=opts)

Then, when attempting to load a website, Chrome will ask me for user + pass, which I can type in manually. That can be avoided by using an extension such as proxy auto auth. After typing in the credentials, by hand or with an extension, the proxy works.

The above approach does not work work when using chrome from SeleniumWire. The proxy-server argument seems to be ignored. The proxy is not set at all. The IP remains unchanged.

Instead I have tried:

from seleniumwire.webdriver import Chrome as WireChrome
proxy_url = f"{user}:{passw}@{domain}:{port}"
seleniumwire_options = dict()
seleniumwire_options.update(
    {
        "proxy": {
            "http": "http://" + proxy_url,
            "https": "https://" + proxy_url,
            "no_proxy": "localhost,127.0.0.1",
        }
    }
)
driver = WireChrome(seleniumwire_options=seleniumwire_options)

That yields me ERR_TUNNEL_CONNECTION_FAILED when trying to connect.

In [377]: seleniumwire.__version__
Out[385]: '4.3.2'

[HMaker]

Try using wrong password to check if selenium-wire complains or browser asks for correct proxy password. If it complains then proxy is being used.

[tsoernes]

It does not complain.

[tsoernes]

The proxy is a NordVPN proxy. Accessing non-SSL content yields 502 Bad Gateway.

[wkeeling]

@tsoernes so just to confirm, HTTPS content works, but HTTP content does not?

[tsoernes]

@wkeeling Neither work, they just give different error messages. I can send you VPN credentials in private so that you can test yourself, if that would help.

[tsoernes]

    from seleniumwire.webdriver import Chrome as WireChrome
    from selenium.webdriver import Chrome, ChromeOptions

    NORDVPN_USER = "XX"
    NORDVPN_PASS = "YY"
    domain = 'de873.nordvpn.com'
    port = 89

    # This works, but you have to enter username + password by hand
    basic_proxy_url = f"https://{domain}:{port}"
    opts = ChromeOptions()
    opts.add_argument("proxy-server=" + basic_proxy_url)
    driver = Chrome(options=opts)
    driver.get("http://www.httpbin.org/ip")
    driver.get("https://www.httpbin.org/ip")

    # This does NOT work
    proxy_url = f"{NORDVPN_USER}:{NORDVPN_PASS}@{domain}:{port}"
    seleniumwire_options = {
            "proxy": {
                "http": "https://" + proxy_url,  # NordVPN support proxy over HTTPS only
                "https": "https://" + proxy_url,
            }
        }
    driver = WireChrome(seleniumwire_options=seleniumwire_options)
    driver.get("http://www.httpbin.org/ip")
    text = driver.find_element_by_tag_name('body').text
    assert text == "502 Bad Gateway\nHttpReadDisconnect('Server disconnected')"
    driver.get("https://www.httpbin.org/ip")
    # Raises:
    # WebDriverException: Message: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
    # (Session info: chrome=91.0.4472.114)

[wkeeling]

OK, reproduced the issue with the credentials you sent through. The issue is also reproducible when running mitmproxy directly outside of Selenium Wire (mitmproxy is what Selenium Wire uses behind the scenes).

Running the mitmdump command (part of mitmproxy):

mitmdump --set mode=upstream:https://de873.nordvpn.com:89 --set upstream_auth=<user>:<pass>

Then configuring the browser to use that (it runs on localhost:8080) results in the same problem.

I checked that the proxy domain de873.nordvpn.com has the proxy_ssl flag set, which it does (I confirmed that by looking in this list: https://nordvpn.com/api/server).

So I'm not sure why the proxy is disconnecting. Is it possible to try other Nord proxies?

[tsoernes]

Ccs

[tsoernes]

I mistakenly closed this. @wkeeling Here is a list of some proxy_ssl compatible proxy servers:

us8815.nordvpn.com
ca1265.nordvpn.com
 jp581.nordvpn.com
 de905.nordvpn.com
us8773.nordvpn.com
 au585.nordvpn.com
  tw41.nordvpn.com
 ie108.nordvpn.com
 se436.nordvpn.com
 se568.nordvpn.com

[Jamie4224]

I also have this issue. I am trying to connect to an authenticated proxy using selenium wire, but chrome asks for authentication credentials. Any fix on this?

[wkeeling]

@Jamie4224 just to clarify how you're specifying the proxy - are you using the proxy-server argument in ChromeOptions or have you specified the proxy using Selenium Wire's own proxy option?

[thatalfredh]

I also have this issue. I am trying to connect to an authenticated proxy using selenium wire, but chrome asks for authentication credentials. Any fix on this?

@Jamie4224 host, port, user, pwd = proxy.split(':') options_seleniumwire = {'proxy': {'http': f"http://{user}:{pwd}@{host}:{port}", 'https': f"https://{user}:{pwd}@{host}:{port}"}} driver = webdriver.Chrome(executable_path=driver_path, options=options, seleniumwire_options=options_seleniumwire)

where options refer to ChromeOptions

[wkeeling]

@thatalfredh have you tried verifying the proxy credentials are definitely correct by setting the proxy manually in Chrome and then browsing to a web page outside of Selenium Wire? Also, does the username or password contain any non-ascii characters?

[goncafab]

I'm finding the same issue. The authenticated proxy works fine with requests the library, but passing it as seleniumwire_options seems to result in net::ERR_TUNNEL_CONNECTION_FAILED.

I've also tried putting the proxy settings as environment variables using HTTP_PROXYand HTTPS_PROXY, and most libraries respect the settings (such as requests), but they also don't seem to work in selenium-wire.

I'm using chromedriver, running headless chrome 91.0.4472.114.

@wkeeling Both username and password contains characters such as -%#$, can this be an issue?

[wkeeling]

@goncafab It's possible that the special characters might be the cause. I'll see if I can reproduce with my own local proxy and a password with those characters in it.

[wkeeling]

Tried reproducing with a local proxy using special characters:

driver = webdriver.Chrome(seleniumwire_options={
    'proxy': {
        'https': 'https://%f$o#o:bar-%#$@localhost:8080'
    }
})

but they don't seem to cause any problem. Is there any other logging information in the terminal @goncafab ?

[goncafab]

@wkeeling Besides the stacktrace, what could I configure to show a better logging output? This is the code I'm using, and the resulting exception:

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-using")

wire_options = {
    'proxy': {
        'http': 'http://myuser%-0000-0000-000-0000000000-name:[email protected]:8080',
        'https': 'http://myuser%-0000-0000-000-0000000000-name:[email protected]:8080',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

browser = webdriver.Chrome(options=chrome_options, seleniumwire_options=wire_options)
browser.get("https://www.google.com/")

Results in:

Traceback (most recent call last):
File "myapp/main.py", line 121, in <module>
     browser.get(url)
   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 436, in get
     self.execute(Command.GET, {'url': url})
   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
     self.error_handler.check_response(response)
   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
     raise exception_class(message, screen, stacktrace)
 selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
   (Session info: chrome=91.0.4472.114)

I'm deploying using docker, but the Dockerfile also works fine locally:

FROM joyzoursky/python-chromedriver:3.8

WORKDIR /apps/myapp

ENV PYTHONPATH "${PYTHONPATH}:/myapp"

RUN pip install --no-cache-dir poetry && poetry config virtualenvs.create false

COPY .flake8 poetry.lock pyproject.toml ./

RUN poetry install --no-interaction --no-ansi

COPY myapp myapp/

CMD ["python", "myapp/main.py"]

[wkeeling]

@goncafab thanks for that extra info. My fear is that we're not going to find a whole lot more info outside of that traceback. The error ERR_TUNNEL_CONNECTION_FAILED often means that the remote end (proxy) terminated the connection for some reason. That said, it might be worth cranking up the log level to debug inside your main.py just in case, e.g.

# Top of script
import logging
logging.basicConfig(level=logging.DEBUG)

[johny322]

Hello, @wkeeling. I have the same problem in my server ubunty, but in my pc windows there is not these problem. In the log, I saw that it would not be possible to connect to the proxy, although everything is fine on the other PC

Its a part of log.

DEBUG:seleniumwire.server:MYSERVERIP: clientconnect
DEBUG:seleniumwire.server:MYSERVERIP: clientconnect
DEBUG:seleniumwire.server:MYSERVERIP: request
  -> HTTP protocol error in client request: Bad HTTP request line: b'GET http://executable_path%3D/usr/bin/chromium-browser HTTP/1.1'
DEBUG:seleniumwire.server:MYSERVERIP: clientconnect
DEBUG:seleniumwire.server:MYSERVERIP: clientconnect
DEBUG:seleniumwire.server:MYSERVERIP: clientdisconnect
DEBUG:seleniumwire.server:MYSERVERIP: clientconnect
DEBUG:seleniumwire.server:MYSERVERIP: serverconnect
  -> ('MYPROXYIP', PORT)
DEBUG:seleniumwire.server:MYSERVERIP: serverconnect
  -> ('MYPROXYIP', PORT)
DEBUG:seleniumwire.server:MYSERVERIP: serverconnect
  -> ('MYPROXYIP', PORT)
DEBUG:seleniumwire.server:MYSERVERIP: request
  -> HTTP protocol error in client request: Bad HTTP request line: b'GET http://executable_path%3D/favicon.ico HTTP/1.1'
DEBUG:seleniumwire.server:MYSERVERIP: clientdisconnect
DEBUG:seleniumwire.server:MYSERVERIP: request
  -> HTTP protocol error in client request: Bad HTTP response line: b'\x00[\x1f@-\x9b\xc8\xdd'
DEBUG:seleniumwire.server:MYSERVERIP: serverdisconnect
  -> ('MYPROXYIP', PORT)
DEBUG:seleniumwire.server:MYSERVERIP: clientdisconnect
DEBUG:seleniumwire.server:MYSERVERIP: request
  -> HTTP protocol error in client request: Bad HTTP response line: b'\x00[\x1f@-\x9b\xc8\xdd'
DEBUG:seleniumwire.server:MYSERVERIP: serverdisconnect

Do you have any ideas? Thanks

[Foxtrod89]

Is that possible to use Token authorization with Nord's SOCKS5?