[🚀 Feature]: Support custom headers for remote WebDriver connection (python)

[scook12]

Feature and motivation

In RemoteConnection, the get_remote_connection_headers method has a set of logical defaults it builds, but there's no way for a user to make an addition to these headers.

Adding a hook or property that the consuming code can set to add to these headers would be a big help.

Usage example

I need to add a host header for requests to be successful to my selenium grid running behind traefik reverse proxy. Without this feature, I can't use selenium python out of the box to create sessions (initializing WebDriver throws 404 errors because traefik doesn't know where to route without the host header).

Passing a parameter from WebDriver to RemoteConnection ought to do the trick:

driver = WebDriver(command_executor, additional_headers={"Host": "some-value"})

[github-actions[bot]]

@scook12, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[symonk]

I think as a workaround you an pass your own RemoteConnection object as a command executor at the moment tho thats a bit heavy handed for the use case maybe, will take a look when i'm home

[jamesmortensen]

Just to clarify, @scook12 are you wanting to set a custom header to allow your test code to access a Selenium Grid on a secured remote server?

For instance, WebdriverIO lets me do something like this in my configuration file:

    port: 443,
    protocol: 'https',
    hostname: 'myremoteseleniumserver.example.com',
    headers: {
        'authorization': 'Bearer some_token_which_the_server_uses_to_keep_out_bad_actors'
    },
    strictSSL: true,
    path: '/wd/hub'

Is something like this what you're looking for, or are you looking to control the headers that the browser sends to the app under test?

[scook12]

@jamesmortensen yes, I need to control headers sent to the grid!

My grid is deployed behind Traefik and using the Host directive requires this header, so my test code needs to pass it to for reverse proxy to route appropriately.

For more context, the intent was to use subdomains to organize my hubs. We test on web and mobile (via appium) deployed on docker swarm. I wanted mobile.domain.com and web.domain.com to route to the appropriate hub service, depending on which platform our test code is being run against.

To do that, I just need to include the Host header in requests sent via WebDriver to the grid. I tested this in a debug session, and it worked, but monkeypatching these headers is really convoluted and creating our own RemoteConnection object is, as symonk mentioned, a pretty heavy solution for a one-line change to the current behavior.

Hopefully that helps clarify!

[jamesmortensen]

UPDATE: Unpublished and renamed to "selenium-proxy" as it's incorrect to say it's a "reverse proxy". Sorry for any inconvenience.

---

@scook12 I have had some success with this Selenium Proxy Server which I just published. I wanted to be able to set the authorization header, but not every test automation library makes that easy to do. With the proxy server, we can modify the headers for any framework we want to use just by pointing the testing library to the proxy server.

So what you can do is configure your testing library to go through this proxy, and then configure the proxy to communicate securely to your Selenium grid. The proxy handles rewriting the headers with the token so the testing library doesn't need to. It's written in Node.js, but that shouldn't be a problem since Selenium WebDriver is a standard. If you try it out, let me know if you run into any trouble.

[titusfortner]

@symonk we could add an additional_headers argument to constructor or as a class property in RemoteConnection to update the default headers

In the meantime the user *could subclass the browser specific subclass, but that is clumsy indeed.

[kaliiiiiiiiii]

Actually, I found a quite sneaky way without using a Proxy or Extension:

You can change the Headers in continue_request() at my_headers = {"sec-ch-ua-platform": "Android"} .

# resource: https://stackoverflow.com/questions/66227508/selenium-4-0-0-beta-1-how-add-event-listeners-in-cdp

class cdp_listener:
    from typing import Dict

    def __init__(self):
        self.listeners = {}
    async def async_helper(self):
        async with driver.bidi_connection() as connection:
            session, devtools = connection.session, connection.devtools

            for listener in self.listeners.items():
                my_listener = await listener[1]["listener"](connection=connection)

                async for event in my_listener:
                    try:
                        await session.execute(listener[1]["at_event"](event=event, connection=connection))
                    except Exception as e:
                        print(e)

    def trio_helper(self):
        import trio
        trio.run(self.async_helper)

    def start_threaded(self, listeners: Dict[str,Dict[callable, callable]] = {}):
        if listeners:
            self.listeners = listeners

        import threading
        thread = threading.Thread(target=self.trio_helper)
        thread.start()
        return thread

    def add_listeners(self, listeners: Dict[str,Dict[callable, callable]]):
        self.listeners = listeners

    def remove_listener(self, listener:str):
        del self.listeners[listener]

async def all_requests(connection):
    session, devtools = connection.session, connection.devtools
    pattern = map(devtools.fetch.RequestPattern.from_json,[{"urlPattern":"*"}])
    pattern = list(pattern)
    await session.execute(devtools.fetch.enable(patterns=pattern))

    return session.listen(devtools.fetch.RequestPaused)

def continue_request(event, connection):
    print({"type":event.resource_type.to_json(),"frame_id": event.frame_id, "url": event.request.url})
    session, devtools = connection.session, connection.devtools

    headers = event.request.headers.to_json()

    my_headers = {"sec-ch-ua-platform": "Android"}
    headers.update(my_headers)
    my_headers = []
    for item in headers.items():
        my_headers.append(devtools.fetch.HeaderEntry.from_json({"name": item[0], "value": item[1]}))

    return devtools.fetch.continue_request(request_id=event.request_id, headers=my_headers)


cdp_listener = cdp_listener()
thread = cdp_listener.start_threaded(listeners= {"continue":{"listener":all_requests,"at_event":continue_request}})

driver.get('https://modheader.com/headers?product=ModHeader')

[github-actions[bot]]

This issue is stale because it has been open 280 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[kaliiiiiiiiii]

pretty sure this would have to be added to the webdriver spec or (if it isn't already) to Webdriver BiDi

[titusfortner]

There's 2 pieces.

• The Remote Connection class uses an http client to send requests and receive responses between the code and the driver/server.
• All assets in the browser are loaded via responses to requests.

BiDi/network interception manages the second one. This issue relates to the first one.

[titusfortner]

Hmm, this might be a part of this larger issue — https://github.com/SeleniumHQ/selenium/issues/12368