apache
Review CI setup and test executions scrips/make targets
Requirement
Make it easy to run e2e tests locally
Problem
As today, a lot CI/testing logic and indirection is hidden in the .github/workflows and .github/actions which make testing and developing quite confusing.
Ideally, it should be possible to easily run almost all the tests executed from the CI locally with minimal configuration.
I would suggest that we wrap all the e2e tests and related setup as Makefile targets
i.e. once checked out, a developer should be able to do something like:
make setup/knative
make test/e2e/knative
The same should happen in the CI pipeline instead of having everything wrapped as an action.
Proposal
No response
Open questions
No response
We already have a structure of tests grouped and executable by kind: https://camel.apache.org/camel-k/2.3.x/contributing/e2e.html#testing-e2e-structure - what we miss is the automation to install Knative, Telemetry or OLM configuration. However if the user has them installed, the test execution should go through correctly.
I saw that, but as stated by the document, they may not work out of the box and the .github/actions is IMHO, quite a mess, so I'd love to see something simpler and that works with minimal setup.
Due to how github actions and scripts are used, running tests locally with make target, even the "corresponding one" to the github action, results in slightly different runs locally or on CI. That can be annoying and time consuming to figure our while debugging issues that might happen on CI and not locally or the other way around.
I'd side with @lburgazzoli on this. IMHO, GitHub Actions should be a test execution client just like myself. If setup is required, a client should be able to this in one-step-per-test-suite (or whatever you like to call the grouping of related tests). GitHub Actions would ideally use the same setup scripts (i.e. with minimal or ideally no difference).
If you like, I could try to give this a shot for the knative tests, since (so far) they all fail for me very reliably ;-)
Just mind that you'd need to take in consideration the situation where, as a user, you already have Knative installed, as a second installation may mess up your local setup.
You can now do ...
./e2e/knative/files/setup.sh
./e2e/yaks/files/setup.sh
If there is nothing else, this can be closed.
You can now also do ...
make setup-knative
make setup-yaks
PR: https://github.com/apache/camel-k/pull/5661