WSS requests are still not fixed

[Avnsx]

A fix is now available in version 4.6.4.

Hey thanks for the quick response! But this does not seem to entirely fix my issue. The website I mentioned seems to load another wss request after the first one and it actually still gets stuck there. Also apart from that all requests are failing saying (failed) net::ERR_CERT_AUTHORITY_INVALID (is it possible they blacklisted the seleniumwire certificate?) or net::ERR_FAILED. This only happens on this specific website for me and only when I have registered a account on the website before and set selenium-wire to use my browser data and browse that website with it in a authenticated session. Would you please help me fix this? 🙏

Originally posted by @Avnsx in https://github.com/wkeeling/selenium-wire/issues/540#issuecomment-1124218414

[BDmitryInoxoft]

The same issue. Summary: In some cases, selenium-wire takes more time to analyze WSS requests (normally- 1.1-1.6 sec, another case - 23-28 sec) and block the next HTTP requests. This leads to a timeout error (page load waits in the Tests) and net::ERR_PROXY_CONNECTION_FAILED in the browser.

Reproduced: 3-5 from 10. Without selenium-wire load time from 1.1 to 1.6 sec 50 from 50 times.

Evrovermant: Python 3.8, PyTest 7.1.2, Selenuim 4.2.0, Google Chrome Version 102.0.5005.61, selenium-wire 4.6.4.

driver setup: options = ChromeOptions() options.add_argument(f'--window-size={Configs.screen_size}') options.add_argument('--incognito') options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') options.add_argument('--disable-infobars') options.add_argument('--disable-extensions') options.add_argument('--disable-browser-side-navigation') options.add_argument('--mute-audio') if Configs.mode == 'headless': options.add_argument('--headless') options.add_argument('--disable-gpu') elif Configs.mode != 'headless' and Configs.mode != 'normal': raise Exception(f'or "{Configs.mode}" is not supported run mode') options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) options.add_experimental_option('prefs', {"plugins.always_open_pdf_externally": True}) logging_prefs = {'browser': 'WARNING'} options.set_capability('goog:loggingPrefs', logging_prefs) s_options = {'disable_capture': True} executable_path = Configs.driver_path service = ChromeService(executable_path=executable_path) driver = webdriver.Chrome(options=options, service=service, seleniumwire_options=s_options) params = {'behavior': 'allow', 'downloadPath': Configs.download_dir} driver.execute_cdp_cmd('Page.setDownloadBehavior', params) driver.scopes = ['webapp./api/']

Additional -

• WSS request outside the scope (webapp./socket/) But it still trying to analyze it.
• Tested without 'disable_capture': True - the same behavior

Questions:

1. Can I somehow add as "exclude_hosts" WSS requests or a part of the URL (like "webapp./wss/req/")? Because all requests have the same host.
2. What else info do you need to debug this?

[BDmitryInoxoft]

@wkeeling If it is not the same issue please notify me, and I will create a separate one.

[Avnsx]

@bdmitryinoxoft This actually sounds more plausible to be my issue, also maybe it's important to note that my issue only exists after selenium wire version 4.6.0, which makes me forced to stay on that version ... But the issue is that I can not use the newest undetected chromedriver version through that sw 4.6.0, so I'm in a pitty here

[BDmitryInoxoft]

Double checked on slenium wire version 4.6.0 - the same behavior

[BDmitryInoxoft]

Just tested it happens even i use only "s_options = {'disable_capture': True}", without driver.scopes

[wkeeling]

@BDmitryInoxoft thanks for the additional info. Can you also tell me which OS you're running on? Also do you have a URL you can share that exhibits the issue?

[BDmitryInoxoft]

@wkeeling OS: Linux - ubuntu 20.04 Windows 11 On the CI - image: python:3.8-slim-buster

Can`t provide a link - the authorization is required. So this is a part of NDA.

The issue just reproduced when I just import selenium-wire and use it. Without any config. Even when i set "s_options = {'disable_capture': True}" I think if there will be a polarisation and one request will now await another this can fix the error.

For debugging, I can try to gather logs. As I remember "driver.requests" - The list of captured requests in chronological order. Can I somehow add a time range to it? This can help find a request where it freezes and I think I can send to you more details.

[BDmitryInoxoft]

@wkeeling

So i try this logger:

            open_user_page()

            wait = WebDriverWait(driver, Configs.delay)
            wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

            for req in driver.requests:
                record = str(f'\n time - {req.date}') + str(f' url - {req.url}')
                try:
                    record = record + str(f' == status code - {req.response.status_code}')
                finally:
                    with open('selenium_log.txt', 'a') as f:
                        f.write(record)

            page.is_opened()

and receive this error on the .js request

                  record = record + str(f' == status code - {req.response.status_code}')
                  AttributeError: 'NoneType' object has no attribute 'status_code'

Can this be a problem?

[BDmitryInoxoft]

update - receive the same error on the API call

[Avnsx]

@wkeeling Could you please maybe introduce a way to just ignore websocket requests in seleniumwires backend entirely when a argument is given? I've huge issues with them and they're holding me back from being able to update from version 4.6.0 ☹️

[BDmitryInoxoft]

Hm. I`m update logger to this

` open_user_page()

        wait = WebDriverWait(driver, Configs.delay)
        wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

        for req in driver.requests:
            record = str(f'\n time - {req.date}') + str(f' url - {req.url}')
            try:
                if req.response:
                    if req.response.status_code:
                        record = record + str(f' == status code - {req.response.status_code}')
            finally:
                with open('selenium_log.txt', 'a') as f:
                    f.write(record)
      
          page.is_opened()

` and receive this on driver.get():

selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 298.706

[BDmitryInoxoft]

looks like something inside the webdriver that we import from seleniumwire

[wkeeling]

@Avnsx Yes this sounds sensible - I'll see what I can do.

Also, if you get a chance, could you try switching a variable in one of Selenium Wire's modules to see whether this improves things? It will mean modifying a file inside your site-packages directory which is probably part of your virtualenv. You can find where your site-packages lives by running pip show selenium-wire.

Once you've found where site-packages lives, the file in question is:

site-packages/seleniumwire/server.py

And the variable is called DEFAULT_STREAM_WEBSOCKETS which you can see near the top of the file. If you change the value to False I would be interested to see whether this makes any difference to your issue.

[Avnsx]

@wkeeling I just first want to say thanks alot for even bothering to help us at all! 👍🏼 I also by exident just figured out how to fix my issue myself, more or less. It seems to have something to do with my chrome user account storage for the website specifically. I went to the top url bar > lock icon > website settings & pressed delete data, then manually logged back in on the selenium session and ever since then it seems to work well for me. Now I am wondering how version 4.6.0 does not have this issue, but then after that version this seems to be an issue for me ... ? I think the website data, that I deleted had some outdated certificate or something that was holding the network requests from passing through on newer seleniumwire versions, because now I'm no longer getting the net::ERR_CERT_AUTHORITY_INVALID error in the developer console (even when starting entirely new). But this "bug fix" kind of takes the sense and purpose behind it; since I was in first place only setting my user data dir to be able to fetch a authentication token (to my logged in user account) for the website through selenium-wire's interception feature. But once removing "all site data" through the menu, I obvisouly am also not logged in anymore. Is there any other way to solve this without clearing everything & logging back into the user account through selenium wire's chromedriver session?

This is the button I am talking about

And also I checked about the server.py file, switching DEFAULT_STREAM_WEBSOCKETS to False did not do any difference for me. Infact I think, the reason only the wss requests were showing up, was because maybe the websocket requests, did not require a specificate certificate verification, which allowed those to pass through.

[BDmitryInoxoft]

@Avnsx Yes this sounds sensible - I'll see what I can do.

Also, if you get a chance, could you try switching a variable in one of Selenium Wire's modules to see whether this improves things? It will mean modifying a file inside your site-packages directory which is probably part of your virtualenv. You can find where your site-packages lives by running pip show selenium-wire.

Once you've found where site-packages lives, the file in question is:

site-packages/seleniumwire/server.py

And the variable is called DEFAULT_STREAM_WEBSOCKETS which you can see near the top of the file. If you change the value to False I would be interested to see whether this makes any difference to your issue.

No effect. Can I set up something, to avoid freezing requests? (ignore errors, pending statuses, unexpected behavior e.t.c)

[wkeeling]

Could you please maybe introduce a way to just ignore websocket requests in seleniumwires backend entirely when a argument is given?

@Avnsx Sorry for the delay on this - been a busy week or so.

So you can ignore websocket requests using the mitm_websocket option which tells the underlying mitmproxy engine to pass the requests through without any processing.

driver = webdriver.Chrome(seleniumwire_options={
    'mitm_websocket': False
})

@BDmitryInoxoft worth also giving that option a try to see whether it resolves your issue.