ecamp
Improve stability of e2e tests
Ways we could improve the stability of e2e tests
Improve the run environment
Thesis: The cypress github action is flaky, the docker images are better
Does not seem that way.
- run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
- run on BacLuc/e2e-use-docker-image: https://github.com/BacLuc/ecamp3/actions/runs/9826692413
Thesis: they are flaky because they don't have enough RAM
Does not seem that way
- run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
- run on BacLuc/e2e-restrict-ram: https://github.com/BacLuc/ecamp3/actions/runs/9826698872
Thesis: they are flaky because they don't have enough CPU
The feeling is that the tests would then just run longer and run into timeouts. But this seems not the case.
Thesis: when we use the cypress retry mechanism, they will be more stable
It does not solve the problem, but renovate can merge and we can work.
- run on upstream/devel with cypress github action: https://github.com/ecamp/ecamp3/actions/runs/9826683335
- run on Bacluc/e2e-use-retry: https://github.com/BacLuc/ecamp3/actions/runs/9826705777
Thesis: the frankenphp container stops responding
(Which leads to the connection refused for the login request errors) -> try enable strace: https://github.com/BacLuc/ecamp3/tree/debug-frankenphp-strace add debug log to caddy: https://github.com/BacLuc/ecamp3/tree/enable-caddy-debug
-> this seems to be confirmed. (See https://github.com/BacLuc/ecamp3/actions/runs/10013887138 in the failing runs in the step added by this PR: https://github.com/ecamp/ecamp3/pull/5558) I have 3 Solutions for this problem, they are listed in these 3 posts:
Vote with a :+1: or :-1:
- https://github.com/ecamp/ecamp3/issues/5493#issuecomment-2240056594
- https://github.com/ecamp/ecamp3/issues/5493#issuecomment-2240057818
- https://github.com/ecamp/ecamp3/issues/5493#issuecomment-2240060761
Improve the single tests
The inspection of the test errors is in this Issue: #5322
Thesis: Not all tests are flaky
- Only clientPrint.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826741651
- Only httpCache.cy.js https://github.com/BacLuc/ecamp3/actions/runs/9826745947
- Only login.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826742342
- Only nuxtPrint.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826740641
- Only persistDashboardFilter.cy.js: https://github.com/BacLuc/ecamp3/actions/runs/9826743440
Thesis: they are flaky because browsers are flaky => use stable browser version Thesis: they are flaky because Cypress is flaky => use Playwright?
Solution 1 for Thesis: "the frankenphp container stops responding"
Revert back to a classic php fpm setup
The thesis is, that the frankenphp is not yet production ready and crashes from time to time. -> Then we should step back to the php fpm setup from before.
Solution 2 for Thesis: the frankenphp container stops responding
Use the prod frankenphp api image for the e2e tests
This seems to work (and hopefully so, we want to deploy this in prod). Needs first #5277 (but is prepared)
Seems to work well: https://github.com/BacLuc/ecamp3/actions/runs/10014833334 But is a little complex:
- https://github.com/ecamp/ecamp3/pull/5560
still crashes, but less often: https://github.com/BacLuc/ecamp3/actions/runs/10014833334/job/27685355483
Solution 3 for Thesis: the frankenphp container stops responding
Add retries in cypress and restart the api container
This also seems to work. Firefox crashes a lot if you use a global beforeEach. @BacLuc would like to remove firefox from the tests for now if we use this approach.
Seems to not work perfectly, but better as before (https://github.com/bacluc-test-org/ecamp3/actions/runs/9998897898):
- https://github.com/ecamp/ecamp3/pull/5561
We still did that, but with cypress retries in
- https://github.com/ecamp/ecamp3/pull/6260
Fix request and assertion order in httpCache.cy.js
- https://github.com/ecamp/ecamp3/pull/6954 or
- https://github.com/ecamp/ecamp3/pull/6953