Selenium Side Runner - Runner hangs when run on Firefox browser

[hep-canoy]

🐛 Bug Report

I am trying to run the test cases (.side) generated from selenium ide using selenium side runner. The tests work perfectly on Selenium IDE, and on Selenium Side Runner with Chrome and Edge browsers, however, when running using Firefox, it will just execute two tests(I have 8 tests in total) and then it will just hang on the RUNS line as you can see in the image below. The "runs" line stopped and produced an error after it met the timeout error (60 * 60* 1000)s. I change it to 10000 ms so that I can see if it will produce any errors.

Same command but with debug mode

These are the errors after

Environment

OS: Windows 10 Selenium IDE Version: Version 3.17.4 Selenium SIDE Runner Version: 4.0.0-alpha.45 Node version: v18.16.0 Browser: Firefox Browser Version: 112.0.2 (64-bit)

[toddtarsi]

@hep-canoy - Oh wow. That almost sounds like maybe the firefox driver is having an issue? I am looking at using Bidi instead of Electron communication apis so I can do local playback on different browsers. I'll keep working on that. Thanks!

[hep-canoy]

Thanks, ill keep myself posted on this matter.

[rafaelarcanjo]

Hi,

Is there any workaround to bypass this error? I'm using the lastest version of selenium-side-runner (today's alpha.58) and Firefox version 118, and I'm encountering the same error scenario.

Thanks.

[toddtarsi]

@rafaelarcanjo - And it only fails on Firefox? Wild, what version of firefoxdriver are you using? I'll join ya (although I'm on Mac so maybe apples and oranges and such)

[toddtarsi]

@rafaelarcanjo - Oh I'm dumb, 118 okay on my way!

[toddtarsi]

@rafaelarcanjo - Before I dive in too far, are any of the known problems on Geckodriver hampering you possibly?

https://github.com/mozilla/geckodriver/releases

[toddtarsi]

@rafaelarcanjo - It's definitely system specific, because I just got this from my attempt to run it:

info: Driver has been built for firefox
info: Finished test If variable Success
 PASS  packages/side-runner/dist/main.test.js (145.221 s)
  Running project Blank
    Running suite New Suite
      ✓ Running test New Test (4858 ms)
      ✓ Running test New Test (4155 ms)
  Running project check-examples
    Running suite checks
      ✓ Running test check-examples (3835 ms)
  Running project Control Flow
    Running suite control flow
      ✓ Running test control flow if (3776 ms)
      ✓ Running test control flow else if (3732 ms)
      ✓ Running test control flow else (4032 ms)
  Running project Echo
    Running suite checks
      ✓ Running test Plugin Commands (3866 ms)
  Running project execute-script
    Running suite execute script
      ✓ Running test execute async script (3775 ms)
      ✓ Running test execute script (3753 ms)
  Running project Execute Script
    Running suite checks
      ✓ Running test execute script array (3957 ms)
      ✓ Running test execute script object (3717 ms)
      ✓ Running test execute script primitives (3778 ms)
  Running project Frames
    Running suite New Suite
      ✓ Running test Frames (4745 ms)
  Running project mouse-events
    Running suite Default Suite
      ✓ Running test mouse sequence (3784 ms)
  Running project nginx-examples
    Running suite inputs
      ✓ Running test input editable (3824 ms)
      ✓ Running test input type checkbox (3798 ms)
      ✓ Running test input type text (3738 ms)
      ✓ Running test select (3765 ms)
    Running suite waits
      ✓ Running test wait implicit (4096 ms)
    Running suite popups
      ✓ Running test popup alert (4124 ms)
      ✓ Running test popup confirmation (3812 ms)
      ✓ Running test popup prompt (4135 ms)
  Running project Plugin
    Running suite checks
      ✓ Running test Plugin Commands (3768 ms)
  Running project select-examples
    Running suite select
      ✓ Running test select-verify-labels (4041 ms)
      ✓ Running test select strategies (3764 ms)
  Running project Select Window
    Running suite Default Suite
      ✓ Running test select window (3780 ms)
  Running project Send Keys
    Running suite checks
      ✓ Running test send keys (4535 ms)
  Running project Simple Parent
    Running suite New Suite
      ✓ Running test open page (3753 ms)
      ✓ Running test outer (5285 ms)
  Running project Store Examples Project
    Running suite store
      ✓ Running test attributes (3741 ms)
      ✓ Running test xpath count (3746 ms)
  Running project Text Comparison Project
    Running suite text
      ✓ Running test text comparison (4074 ms)
  Running project Simple
    Running suite 2A - Suite 1
      ✓ Running test Test 1 (3908 ms)
    Running suite 2B - Suite 2
      ✓ Running test Test 2 (4008 ms)
  Running project New Project
    Running suite New Suite
      ✓ Running test Times variable (3993 ms)
      ✓ Running test If variable (3733 ms)
  Running project Wait For Project
    Running suite wait for
      ✓ Running test wait for element present (1 ms)
      ✓ Running test wait for element not present (1 ms)
      ✓ Running test wait for element visible (2 ms)
      ✓ Running test wait for element not visible (1 ms)
      ✓ Running test wait for element editable (and not editable) (1 ms)

Test Suites: 1 passed, 1 total
Tests:       41 passed, 41 total
Snapshots:   0 total
Time:        145.252 s, estimated 162 s

Please let me know if I can help in any way if thats the case. Good luck!

[rafaelarcanjo]

@toddtarsi I use Selenium Grid, running the Node Docker (selenium/node-docker:4.5.3), with Google Chrome version 115.0 (selenium/standalone-chrome:115.0), and Firefox version 116.0 (selenium/standalone-firefox:116.0). The error only occurs in Firefox; it runs successfully in Chrome. Thanks.

[toddtarsi]

@rafaelarcanjo - That sounds like a "known problem" of Geckodriver, specifically:

Startup hang with Firefox running in a container (e.g. snap, flatpak):

When Firefox is packaged inside a container (like the default Firefox browser shipped with Ubuntu 22.04), it may see a different filesystem to the host. This can affect access to the generated profile directory, which may result in a hang when starting Firefox. Workarounds are listed in the geckodriver usage documentation.

[rafaelarcanjo]

@toddtarsi The strange thing is that the test completes successfully, the session is started, the test is executed, and the session is terminated. The success message is displayed in the console, but the site runner doesn't finish.

[toddtarsi]

@rafealarcanjo - Oh, I see what you mean from the logs. Thanks for all the screencaps and info there, perhaps I should have looked closer at those at first. Hm. There's another set of logs you can access by doing something like this:

NODE_DEBUG="webdriver" selenium-side-runner [rest of command]

This will log the internals of the running webdriver process. Can you try that and post your log dump here or just use the selenium slack channel to direct message it to me (Todd Tarsi) if you're nervous about the contents?

[rafaelarcanjo]

Hi @toddtarsi I'ii try to explain. On the right side is the executed command, I've hidden some details, and on the left side is the Selenium grid

As you can see, the session has been created, and the test is currently running.

Now, in the following image, you can verify that the session has been removed, and the message indicating that the test run successfully is displayed

I'm sending the log using the "-d" flag since the variable NODE_DEBUG="webdriver" didn't make a difference. https://pastebin.com/ZN0HxUsU

Thanks!

[Dylan700]

Any updates on this @toddtarsi @rafaelarcanjo? I'm experiencing the same issue, all my tests finish successfully, but it hangs indefinitely at the end.

I'm using Docker to run the tests using Mozilla Firefox 120.0.1 and geckodriver 0.33.0.

Here is the debug output I get before it hangs:

Running tests for Firefox
info: Running test Click Search Box
debug: Playing state changed prep for test Click Search Box
info: Building driver for firefox
info: Driver has been built for firefox
debug: Playing state changed playing for test Click Search Box
debug: executing open|/
debug: passed open|/
debug: executing setWindowSize|1487x892
debug: passed setWindowSize|1487x892
debug: executing click|id=APjFqb
debug: passed click|id=APjFqb
debug: executing close
debug: passed close
debug: Playing state changed finished for test Click Search Box
info: Finished test Click Search Box Success

And here is the command I use to start the tests: MOZ_HEADLESS=1 JEST_JUNIT_OUTPUT_DIR=$GITHUB_WORKSPACE/reports JEST_JUNIT_OUTPUT_NAME=selenium-firefox-tests-report-$FILE_NAME.xml selenium-side-runner -c "browserName=firefox moz:firefoxOptions.args=[-headless, -acceptInsecureCerts, -safe-mode, -no-remote]" -z ./screenshots --retries 1 --output-directory ./reports -j " --reporters=jest-junit --reporters=default " --timeout 5000 --debug --base-url https://google.com *.side

I think the -safe-mode arg has something to do with it. Although, it seems I must include this otherwise I get the following error:

 ● Running project Google › Running suite Default Suite › Running test Click Search Box

    SessionNotCreatedError: Error: NS_ERROR_UNKNOWN_HOST

      at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:524:15)
      at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:601:13)
      at Executor.execute (node_modules/selenium-webdriver/lib/http.js:529:28)