kiwix-js icon indicating copy to clipboard operation
kiwix-js copied to clipboard

Experiment with adding "latest" to saucelabs config

Open Jaifroid opened this issue 1 year ago • 3 comments

Experiment to fix #894.

Jaifroid avatar Oct 08 '22 09:10 Jaifroid

Adding "latest" doesn't seem to solve the issue.

image

Jaifroid avatar Oct 08 '22 09:10 Jaifroid

We are getting "InfrastructureError"....

2022-10-08T09:47:21.838Z ERROR webdriver: Request failed with status 500 due to Error: Infrastructure Error -- The Sauce VMs failed to start the browser or device.
For more info, please check https://docs.saucelabs.com/dev/error-messages
2022-10-08T09:47:21.840Z ERROR webdriver: Error: Infrastructure Error -- The Sauce VMs failed to start the browser or device.

Jaifroid avatar Oct 08 '22 09:10 Jaifroid

Firefox 104 (one before latest) is producing same error. Strange that Firefox 45 can run, but this one can't...

image

Jaifroid avatar Oct 08 '22 10:10 Jaifroid

I've found some info on the error we're experiencing: https://docs.saucelabs.com/dev/error-messages/index.html#the-sauce-labs-virtual-machine-failed-to-start-the-browser-or-device. The salient informaiton is:

Causes

  • If you're running a Selenium test, this usually means that you're specifying a Selenium version that isn't compatible with the browser/version/OS you selected.
  • If you're running an Appium test, this usually means that you're specifying a Appium version that isn't compatible with the browser/version/OS you selected.
  • You may have requested an incompatible platform/OS/browser combination.

I assume our issue is the first or last of these bullet points.

Jaifroid avatar Oct 27 '22 08:10 Jaifroid

I've disabled Firefox latest, which allows us to get past the QUnit tests and on to the end-to-end tests. But now we're getting this persistent failure with both the latest Nightwatch and with the old Nightwatch we used previously:

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (_http_outgoing.js:[47](https://github.com/kiwix/kiwix-js/actions/runs/3392039979/jobs/5637806528#step:7:48)0:11)
    at module.exports.ResponseStream.(anonymous function) [as setHeader] (/home/runner/work/***-js/***-js/node_modules/union/lib/response-stream.js:100:34)
    at Object.exports.(anonymous function) (/home/runner/work/***-js/***-js/node_modules/http-server/lib/core/status-handlers.js:57:7)
    at Readable.stream.on (/home/runner/work/***-js/***-js/node_modules/http-server/lib/core/index.js:339:22)
    at Readable.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

Jaifroid avatar Nov 04 '22 07:11 Jaifroid

Same error with setting headers after they are sent to client persists even with the original configuration minus Firefox latest.

@kelson42 Having exhausted my knowledge about this configuration, and having made 24 commits to this PR tweaking various options, I can only conclude that a fundamental API change has occurred, and that we have a hopelessly out of date configuration here in accessing Sauce Labs. It needs re-building from the ground up, and if I'm going to do that work, it might as well be with a more modern testing infrastructure.

Ideal situation would be to complete #554 -- mossroy and I had discussed using the modern framework vite.js for this, and it has a test suite called https://vitest.dev/. I'd need to look at how it integrates with: 1. SauceLabs and 2. BrowserStack.

But this would be a major job in which I have extremely limited experience.

So maybe if I can get a new config running on BrowserStack by following their step-by-step instructions, which look very thorough, it would be the best compromise for now, along with retaining the QUnit tests that do run successfully in this configuration.

Jaifroid avatar Nov 04 '22 08:11 Jaifroid

So we can get on with daily business I have had no choice but to disable the end-to-end tests, and to modify the cross-browser QUnit tests so they do not run the latest Firefox. We still have latest Firefox tested in GitHub actions, so this is no real loss, but disabling end-to-end UI testing is a loss, which I hope we can fill with a BrowserStack configuration.

For now, we still have:

  • QUnit tests run in Firefox and Chrome headless on GitHub actions, for all contributors.
  • QUnit tests run on a suite of Edge (Chromium and Legacy), and Chrome (old and new), and Firefox (old) on SauceLabs.

Jaifroid avatar Nov 04 '22 08:11 Jaifroid