e2e tests example
requested in https://github.com/AlexZzz/libvirt-exporter/issues/30
this is draft PR with e2e tests example based on VMware Fusion + vagrant + ansible + molecule. One of best advantage of this solution that this is local first e2e tests. But downside of this local first solution that this is incompatible with current CI (circleci).
With this PR it's possible to open discussion about e2e test for this project. I'm open to all comments)
Current env consist of:
- ubuntu2010 vm (base)
- installed libvirtd, podman inside vm
- created vm (cirros) inside base vm (ubuntu2010)
- cloned project sources inside base vm
- built libvirt_exporter images and run libvirt_exporter container
e2e tests located in ansible/molecule/default/verify.yml
Thanks for the PR!
Unfortunately, I didn't have time to check how it works right now. But it seems like a good starting point.
I see two huge problems with e2e tests of this exporter. They come from it's essense:
- This exporter requires libvirt daemon and virtual machine (or machines?) running. It also requires access to libvirtd socket available. I don't know any public CI solution with nested virtualization available.
- In the best-case scenario tests should validate metrics. As a reference, we can take values from
virshcommand output. Ifvirshstatistics shows the same results as libvirt-exporter (don't forget possible units change here! Ex: nanoseconds -> seconds, etc.) then tests seem passed. But how is that possible to avoid values drift which might (and will) happen betweenvirshcommand run and libvirt-exporter's metrics collection?
For the first problem: we may run a virtual machine in some cloud via API. Connect to that machine, start nested vm and run tests. I'm not sure how adequate this is. Seems like a lot of hard work, there's no need in such automation right now.
Also, I've heard that circle-ci will ban me if I log into it from Russian IP:) don't want to check it, but it's possible to mistakenly do it and ruin all the CI and tests.
All these complexity makes me think we don't need a CI-based tests for this tiny project. Local-only tests are more than OK. But, it's still interesting how is it possible to validate metrics values, maybe there's some easy-to-implement solution for this?