canonical
[REQ] Guidance for functional testing
It would be helpful to include some documentation/guidelines/best practices for functional testing. Harness is great for unit testing, but functional testing for charms remains a bit nebulous.
Right now, there's basically 2 choices:
- Zaza testing framework. Not sure if this supports operator testing (but I imagine it should be fine)
- Just using python-libjuju and creating your own custom quasi-framework.
I think operator charming framework should provide some sort of guidance in the docs for doing functional testing.
+1 to this.
Using Zaza would be nice, as we'd be coordinating w/ the way that the Canonical OpenStack team does things. I'd prefer that we take that direction, if possible.
I'd add that the Kubeflow team have done some good work with their pytest-operator, examples of usage can be found here and here.
I think I like this route, as it's a little less Openstack-y for people who aren't writing Openstack charms. The Kubernetes, Kubeflow and Observability teams have gone this route, we're currently just working out what feels like best practice it.
Certainly seems like something this team would contribute to in the documentation though.
Just my two cents - as someone who was initially leaning against Zaza for use within the BootStack team, and eventually came around to support it, I'm +1 for Zaza.
What I really like about Zaza is that it totally decouples model creation from tests. If I have to debug an issue, I can have my model set up once, and then I can go in and just re-run the tests without a full model redeploy (assuming the tests don't have side effects preventing this). This can be a significant timesaver, especially for non-container-based charms. Zaza does have a learning curve, and perhaps energy should be spent in making that easier, but I think it's a good starting point.
That being said, I think it's certainly worth considering alternatives. I'm curious on what pytest-operator provides, for example, but simply haven't been exposed to it yet.
We're planning to look at improving our testing story and documentation in the next cycle, including integration testing. Keeping this open till then.
A doc with significant guidance has been added here: How to write integration tests for a charm. In addition, integration tests are part of the K8s charm tutorial: https://juju.is/docs/sdk/write-integration-tests-for-your-charm