ecamp
Tests on CI run slow, too often and flaky
This is a collection of issues I see with our current CI setup that make development slow and makes the life of renovate hard.
Problem description
- Generally, our tests take a rather long time (~15min in average)
- The tests run often, sometimes testing the same commit that was already tested before
- Our e2e tests are still flaky, leading to necessary repetition of tests
- Higher utilization of workflow runners
- PRs getting kicked out of merge queue (=queue has to restart other jobs)
- Renovate not being able auto-merge => high clutter in PR backlog
- Fixing a bug and deploying it to prod takes a huge time (as we saw during the last hackathon). CI needs to run min. 6 times successfully:
- First, CI needs to run on PR (twice in parallel)
- Then CI needs to run on merge queue
- After merging a PR from merge queue, dev again runs CI (although the same commit was already tested in merge queue)
- When fast-forwarding to stage & prod, the CI runs again (although the same commit was already tested before)
- Renovate still seems to have a problem with our setup: https://github.com/ecamp/ecamp3/issues/4309 This might be unrelated to above. I couldn't figure out anything on renovatebot issues list. I wonder though if this is somehow connected to merge queue and the config rebaseWhen:conflicted.
Potential solutions (to be discussed)
- Challenge benefit of merge queue (does it bring the benefits we hoped?)
- Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.
- Somehow making API tests faster
- When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?
- Find a way to avoid running the same CI for a commit hash where CI has already run successfully before
- other ideas?
Challenge benefit of merge queue (does it bring the benefits we hoped?)
I quite like it. Especially when many people are actively pushing/merging
Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.
I think we should try to make the e2e test less flakey. I don't like not testing them, as before we upgrade to vue3 I'd like add more tests. We could try to use https://docs.cypress.io/guides/continuous-integration/github-actions#Testing-with-Cypress-Docker-Images instead of pulling the latest one.
Somehow making API tests faster
The question is, are we testing the right way?
When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?
That was once the plan.
Find a way to avoid running the same CI for a commit hash where CI has already run successfully before
We could try to run e.g. the api test on PR push only if certain conditions match?
Core Meeting Decision
Things we want to tackle:
- [x] Try to use cypress image https://github.com/ecamp/ecamp3/pull/5414
- [x] Improve Stability of cypress tasks
- [x] Improve Stability of api tests (https://github.com/ecamp/ecamp3/pull/5292)
- [x] Reduce duration of api tests (maybe parallelism)
- Happened without our doing: https://github.com/sebastianbergmann/phpunit/pull/5861
- Further reduction of CI time: https://github.com/ecamp/ecamp3/pull/5372
- [x] Increase the duration of the performance check. We would like to have a benchmark, because the CI is not guaranteed to run stable or performant
- [ ] ~~Create reporting of certain dev endpoints under repeatable circumstances~~ Currently we are more or less good with monitoring the performance in prod, testing against prod data locally and checking the performance on staging.
- Local E2E use separate DB
- -> https://github.com/ecamp/ecamp3/issues/7909
Maybe/explore:
- Only run stable e2e tests
- skip duplicate actions
