sandstorm

sandstorm copied to clipboard

So, I noticed tests were broken yesterday, and the only error we got was hilariously because there were no screenshots to upload. I'm guessing this is the error in question though:

 Connection refused! Is selenium server started?
{
  value: {
    message: 'session not created: This version of ChromeDriver only supports Chrome version 90\n' +
      'Current browser version is 89.0.4389.114 with binary path /usr/bin/google-chrome\n' +
      '  (Driver info: chromedriver=90.0.4430.24 (4c6d850f087da467d926e8eddb76550aed655991-refs/branch-heads/4430@{#429}),platform=Linux 5.4.0-1043-azure x86_64) (WARNING: The server did not provide any stacktrace information)\n' +
      'Command duration or timeout: 5.43 seconds\n' +
      "Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'\n" +
      "System info: host: 'fv-az235-407', ip: '10.1.1.93', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-1043-azure', java.version: '11.0.10'\n" +
      'Driver info: driver.version: unknown',
    error: 'session not created'
  },
  status: 33
}

#3514 will prevent tests from causing an error when screenshots are not uploaded. I believe this is the correct choice, because when we have no test failures, we should have no screenshots. However, we need to address that the main testing step can succeed even if tests fail to run entirely.

Interestingly, https://github.com/sandstorm-io/sandstorm/runs/2463490040?check_suite_focus=true tests are working again, it seems the runner has been updated, but we really should figure out how to have this not break for a number of days every time a new Chrome version comes out.

tests/account and apps/web-publishing are the only two tests that failed in the above run.

Is there an actionable task associated with this issue? Should we close it?

Yep, realized this was older than I thought.

This is happening again, and I suspect we can do something to make it better. I am not positive how/where/when we install both Chrome and ChromeDriver in our CI, but I think we may need to figure out how to get them from the same source (so they are appropriately synced) or update Chrome somehow. I am doing a test or two on a branch to see if I can fix it while it's happening again.

https://github.com/sandstorm-io/sandstorm/commit/6d881327aeea4d57941e240bd65091bb40f9b4ff (and https://github.com/sandstorm-io/sandstorm/commit/6d881327aeea4d57941e240bd65091bb40f9b4ff) are my best theory on how to fix this (ignore all the other garbage commits on that branch, I am going to delete it when I have a solution that builds.

So to document what I tried here, as the runner has updated and this is no longer really replicable for the moment:

in run-local.sh I tried CHROME_VERSION=$(google-chrome --version | awk '{print $3}') and node_modules/.bin/selenium-standalone install --drivers.chrome.version=$CHROME_VERSION to ensure we grabbed a ChromeDriver that matched the Chrome version the runner had.

I got an error stating that it wasn't the latest version or something similar? This seems to suggest ensuring we use the runner's built-in ChromeDriver also may not work with CI here. Also, Ian pointed out his machine uses chromium and chromium-browser may also be used some places, which means this solution would break local tests.

It may be preferable to manually ensure we are updating google-chrome in the runner from an external source so it matches the latest version used by our tests. A plain apt-get command set did not do this, as the source appears set to use whatever GitHub is currently shipping on their runner.