libvirt-exporter icon indicating copy to clipboard operation
libvirt-exporter copied to clipboard

e2e tests example

Open Ubun1 opened this issue 3 years ago • 1 comments

requested in https://github.com/AlexZzz/libvirt-exporter/issues/30

this is draft PR with e2e tests example based on VMware Fusion + vagrant + ansible + molecule. One of best advantage of this solution that this is local first e2e tests. But downside of this local first solution that this is incompatible with current CI (circleci).

With this PR it's possible to open discussion about e2e test for this project. I'm open to all comments)

Current env consist of:

  • ubuntu2010 vm (base)
  • installed libvirtd, podman inside vm
  • created vm (cirros) inside base vm (ubuntu2010)
  • cloned project sources inside base vm
  • built libvirt_exporter images and run libvirt_exporter container

e2e tests located in ansible/molecule/default/verify.yml

Ubun1 avatar Apr 08 '22 14:04 Ubun1

Thanks for the PR!

Unfortunately, I didn't have time to check how it works right now. But it seems like a good starting point.

I see two huge problems with e2e tests of this exporter. They come from it's essense:

  1. This exporter requires libvirt daemon and virtual machine (or machines?) running. It also requires access to libvirtd socket available. I don't know any public CI solution with nested virtualization available.
  2. In the best-case scenario tests should validate metrics. As a reference, we can take values from virsh command output. If virsh statistics shows the same results as libvirt-exporter (don't forget possible units change here! Ex: nanoseconds -> seconds, etc.) then tests seem passed. But how is that possible to avoid values drift which might (and will) happen between virsh command run and libvirt-exporter's metrics collection?

For the first problem: we may run a virtual machine in some cloud via API. Connect to that machine, start nested vm and run tests. I'm not sure how adequate this is. Seems like a lot of hard work, there's no need in such automation right now.

Also, I've heard that circle-ci will ban me if I log into it from Russian IP:) don't want to check it, but it's possible to mistakenly do it and ruin all the CI and tests.

All these complexity makes me think we don't need a CI-based tests for this tiny project. Local-only tests are more than OK. But, it's still interesting how is it possible to validate metrics values, maybe there's some easy-to-implement solution for this?

AlexZzz avatar Apr 24 '22 14:04 AlexZzz