Cypress test sometimes fail on local but not on Vercel or vice versa

[gsambrotta]

Bug description On branch https://github.com/kendraio/kendraio-app/pull/451, tests were running all green on local but when pushed to GitHub, Vercel tests (which have a different set-up than the one in package.json scripts) had 1 test failing. Specifically:

/kendraio-app/cypress/e2e/context_and_state.ts 
line 232 - 'should allow the disabling of the datagrid'

After running the test on local again, I confirmed this time this test was also failing on local. At that point, Vercel was showing no test failing. Without pushing new code. I pushed a "fix" for the test which show all green on local and showed many tests failing on Vercel. After checking again on local, I noticed the test were passing with one script (run without visual Electron window open) but not with another script.

After reverting the fix to the original state, the above-mentioned test was failing with one script in local, was not failing with another script and all the tests were passing on Vercel.

This behaviour seems apparently random and could be just about this branch, but I am sharing my experience here in case it will happen again. To have a history of this issue.

We should also look into syncing Vercel and local way of setting up and running Cypress tests.

Please check https://github.com/kendraio/kendraio-app/pull/451 comments and commits test history to understand better.

Reproduction steps

1. Go to branch revert-fow-instead-workflow
2. Run: "e2e:serve_and_retest_on_change" -> Check if all test are green
3. Now run: "e2e:autorun" -> Check if result is the same as previous step
4. Check in particular test: Context and State
5. If everything is OK, try the above steps again at this commit history: https://github.com/kendraio/kendraio-app/pull/451/commits/34e86db5f0441e5c8c3886959d284d1cadd29b7e

Expected behaviour If a test fail, it should fail on local and in Vercel. And it should fail regardless of which scripts I run to run the tests.

Runtime environment

• OS: Ubuntu 24.04
• Browser: Firefox
• Cypress version: "cypress": "^13.3.3"
• Angular Version: 16

[gsambrotta]

The same is happening on this branch, specifically here: https://github.com/kendraio/kendraio-app/actions/runs/8143928730/job/22256881189?pr=454

It might be related to Angular 15 to 16. Or maybe not as they are not failing on https://github.com/kendraio/kendraio-app/pull/425

Seems that the script: e2e:autorun do not show the failing test even if e2e:serve_and_test do show it. With the new scripts test:autorun and test:serve_and_test the results seems consistent between different scrips, in local. Not sure about Vercel but seems consistent there too.

[gsambrotta]

In PR #454 there is an anusual behaviour: At first the Vercel show a failing test, which is stil visible in the last commit: https://github.com/kendraio/kendraio-app/actions/runs/8143928730/job/22256881189 But then after some minutes, Vercel show all test passing, without changing any code or pushing new code.

What I would do now is:

1. Be sure the scripts in all the branches are up-to-date with new scripts (starting with test)
2. Be sure that also the scripts in Vercel are run the same scripts (starting with test)
3. Be sure local and Vercel are synced
4. Observe if it's happening again