chainsaw: added poc test

[akselleirv]

Relates to https://github.com/flux-iac/tofu-controller/issues/1131

This is a rewrite of the main parts of the controllers/tc000010_no_outputs_test.go test to showcase the benefits of Chainsaw.

Resources can be reused across the tests by placing the YAML manifests in a shared folder, instead of inlining them like I have done here. I find the declarative approach to testing much more readable compared to the more imperative Go test.

The current test uses oras to pack and push the OCI image to the local kind registry, but can be replaced with for example the flux CLI if we would like to avoid the dependency.

The test can be run with the following commands:

./tools/reboot.sh 
tilt up
chainsaw test test/no-outputs/

Is this approach something that could replace parts of the current controller test suite? Or would it be only be suitable for replacing the e2e tests?

[chanwit]

The test structure is so clean and very easy to understand. Thank you so much @akselleirv

What do you think of putting a test number as the prefix of dir no-output?

We found that test case numbering gave us a good way to organizing, grouping while keeping the flat structure of test cases at the same time.

[chanwit]

Since @yitsushi spent a load of time restructuring tests in CI, I'd to hear your options too.

[akselleirv]

What do you think of putting a test number as the prefix of dir no-output?

I can definitely do that! That makes it easier to run them in parallel groups like you have done for the existing tests.

[yitsushi]

I'm not against chainsaw itself, but in my opinion it should not replace go tests.

For e2e is a different question that's totally fine, but controllers/tc* are not e2e, they are (supposed to be) unit tests.

---

A totally opinion based, 100% personal take on the topic: Speaking of myself, I would not write tests in this format. Nothing special, it's just not how I would work on any projects and if a project would require me to write yaml files as test, that would discourage me to even think about contributing to that project. I'm well aware it's a personal issue, but I can't help that. I like when I have the least amount of abstraction on top of the tools I'm using. As I write Go code, I expect Go tests, a test code that can be changed easily if required. Abstraction layers make it much harder to change things, they force the user in a give way and "deal with it" if that doesn't fit. I'm totally find adding extra tests, but I prefer to have go tests for everything that's written in go. To remove the ugliness of inline yaml files, there are other options (for example using fixtures directory, a module with specific structure, using embed, etc).

a quick summary: if I have to update a test or write a test in a yaml file, I would be very discouraged to even think about contributing to a project, and I would need a very strong reason why I have to (not why I want to) contribute.

[akselleirv]

I definitely see your point @yitsushi!

Maybe start by replacing the current e2e test workflow to chainsaw. I think it’s easier to read and then the e2e test can be executed locally instead of having to push it to GitHub in order to trigger the workflow.

Then later on extract the more e2e go tests within the controllers/ to be using Chainsaw?

[ilithanos]

I actually agree a lot with @yitsushi on this one.

regarding e2e test they can already be run locally using the shell script inside the repo, so running them locally is already a fixed problem.

I do like the syntax for testing e2e though with chainsaw, but unit tests i would keep as go tests.

I personally prefer a toolchain to be as close to the language standards for the language i'm developing in as possible when writing tests, to prevent context switching where possible. Chainsaw seems like a good tool for testing yaml deployments and stuff like that where you are already writing yaml.

So TLDR:

Keep unit tests as go tests as it's more idiomatic etc. E2E tests might be a good fit for chainsaw tests, but the local testing argument is not really a real issue currently. So it would need to be done for readability and maintainability as main reasons.

[akselleirv]

I'll close this PR and then we can consider using Chainsaw for e2e tests in the future. Thanks for all the feedback!