Story: a new charm author tries to run charm tests for the first time.

[pengale]

This is an exercise where I make an effort to be a "naive" user, coming to our framework for the first time.

I assume that at baseline, I:

1. Know my way around the Ubuntu CLI.
2. Know how to use git to clone a repository.

Here's how things went.

0. I spin up a fresh Ubuntu VM with Multipass, and connect to it. This is part of our baseline assumption, and I did not attempt to act in a naive way here. I'm only trying to be "naive" when it comes to Juju and Operator Framework specific things.

1. Run tox

Our docs currently talk about run_tests, but we are moving toward tox. We run into our first issue here: Tox isn't installed by default. The error message, which is built into Ubuntu is friendly:

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ tox

Command 'tox' not found, but can be installed with:

sudo apt install tox

2. Following the instructions in the error message, I run apt install tox

This fails:

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo apt install tox
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package tox

This was due to me not yet having run apt update on the fresh VM. We're assuming that I'm an experienced Linux user, so I went ahead fixed it without guidance:

sudo apt update && sudo apt install tox

3. Run tox

This works, and the unit tests run. That's a lot of progress! I could switch gears to writing a charm from here. But what if I want to run integration tests?

4. The doc I'm writing instructs authors to run tox -e integration.

This fails, with an error message about dependencies:

ERROR tests/integration/test_defer.py::test_smoke - RuntimeError: Missing dependencies: juju, charmcraft
====================================================================== 2 warnings, 1 error in 0.04s ======================================================================
ERROR: InvocationError for command /home/ubuntu/charm-defer-with-dispatch/.tox/integration/bin/pytest -v --tb native --ignore=/home/ubuntu/charm-defer-with-dispatch/tests/unit --log-cli-level=INFO -s (exited with code 1)
________________________________________________________________________________ summary _________________________________________________________________________________
ERROR:   integration: commands failed

5. I need to install dependencies.

I just installed something with apt, so that's where my head is at. With no explicit guidance from the error message on next steps, I try the tools I have at hand:

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo apt install juju
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package juju

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo apt install charmcraft
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package charmcraft

Oh dear. These error messages are identical to the ones that I fixed by running apt update! That's unfortunate. It's also outside of the scope of things that we can affect with the Framework.

What we can do is to document this flow, on Discourse, and in CONTRIBUTING.md. Those docs should tell us to install the snaps.

6. Install the Juju snap.

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo snap install juju
error: This revision of snap "juju" was published using classic confinement and thus may perform
       arbitrary system changes outside of the security sandbox that snaps are usually confined to,
       which may put your system at risk.

       If you understand and want to proceed repeat the command including --classic.
ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo snap install juju --classic
juju 2.9.28 from Canonical✓ installed

This works, after I read the error message, and pass the --classic flag.

7. Install Charmcraft

The same thing happens with Charmcraft -- all works, so long as I pass --classic:

ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo snap install charmcraft
error: This revision of snap "charmcraft" was published using classic confinement and thus may
       perform arbitrary system changes outside of the security sandbox that snaps are usually
       confined to, which may put your system at risk.

       If you understand and want to proceed repeat the command including --classic.
ubuntu@enormous-pony:~/charm-defer-with-dispatch$ sudo snap install charmcraft --classic
charmcraft 1.5.0 from Canonical✓ installed

8. With those steps completed, let's see if I can run the tests.

I run tox -e integration. Oh dear, this error message is a little obscure:

ERROR tests/integration/test_defer.py::test_smoke - FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.local/share/juju/controllers.yaml'
====================================================================== 2 warnings, 1 error in 0.03s ======================================================================
ERROR: InvocationError for command /home/ubuntu/charm-defer-with-dispatch/.tox/integration/bin/pytest -v --tb native --ignore=/home/ubuntu/charm-defer-with-dispatch/tests/unit --log-cli-level=INFO -s (exited with code 1)
________________________________________________________________________________ summary _________________________________________________________________________________
ERROR:   integration: commands failed

The meaning of No such file or directory: '/home/ubuntu/.local/share/juju/controllers.yaml' is that we don't have a controller bootstrapped. But you have to know some Juju internals to understand why that is the correct translation of the jargon.

We need to document this step super well!

9. Let's try bootstrapping: juju bootstrap localhost

This Just Works, which is wonderful. I didn't know that we could bootstrap an lxd cloud without running lxd init --auto first. :-)

10. Try the tests again!

I run tox -e integration.

Uh-oh. There are some dense error message, but the short of the story is that I have run out of space in my virtual machine, and I can't run charmcraft pack.

At this point, I ran out of the time that I allocated to this task. I am 99% certain that things would work were I to resize the disk, or just spin up a machine with a bigger disk. We should add minimum system requirements to the docs, though!

[Abuelodelanada]

Hi @pengale !

Some time ago @sed-i from the Obsevability team created a great Multipass workflow called charm-dev that will make life easier for charm authors (I use it every day and it's great!)

$ multipass find
Image                       Aliases           Version          Description
snapcraft:core18            18.04             20201111         Snapcraft builder for Core 18
snapcraft:core20            20.04             20210921         Snapcraft builder for Core 20
snapcraft:core              16.04             20210929         Snapcraft builder for Core 16
core                        core16            20200818         Ubuntu Core 16
core18                                        20211124         Ubuntu Core 18
18.04                       bionic            20220419         Ubuntu 18.04 LTS
20.04                       focal,lts         20220419         Ubuntu 20.04 LTS
21.10                       impish            20220309         Ubuntu 21.10
daily:22.04                 devel,jammy       20220420         Ubuntu 22.04 LTS
appliance:adguard-home                        20200812         Ubuntu AdGuard Home Appliance
appliance:mosquitto                           20200812         Ubuntu Mosquitto Appliance
appliance:nextcloud                           20200812         Ubuntu Nextcloud Appliance
appliance:openhab                             20200812         Ubuntu openHAB Home Appliance
appliance:plexmediaserver                     20200812         Ubuntu Plex Media Server Appliance
anbox-cloud-appliance                         latest           Anbox Cloud Appliance
charm-dev                                     latest           A development and testing environment for charmers
docker                                        latest           A Docker environment with Portainer and related tools
minikube                                      latest           minikube is local Kubernetes

[sed-i]

Thanks @Abuelodelanada! Just for reference: all the necessary steps until tox -e integration are documented in the cloud-init script of that multipass blueprint. Similar dev-env setups are listed here.

[pengale]

@sed-i @Abuelodelanada thanks for the comments and links.

The intent of this issue was to see what happened if someone sat down and tried to run tests after checking out a charm, without referencing any of the additional documentation and helpers that we've created.

I'm interested in the experience of someone outside of Canonical, who is not yet enfranchised, but is interested in charming. Perhaps they read an article on Hacker News, or a blog post.

If that experience is negative, full of confusing error messages and potential dead ends, it is unlikely that the person will reach out to us to learn more. Part of our goal as a team should be to make our tools work as well as possible for a naive user. We can build more tooling and appliance that make the process even better, but at baseline, someone should be able to stumble their way to doing things like running tests, or doing a test deployment.

Does that make sense?

[pengale]

This ticket can be closed when:

1. Test dependences are documented in CONTRIBUTING.md and in appropriate places in the Discourse docs
2. The same docs point to advice on bootstrapping a juju controller
3. The docs above document base dev environment requirements
4. Pytest-operator error messages include helpful suggestions on failure to find a Juju controller, and failure to charmcraft pack (e.g., it might link to the Juju bootstrap docs, and suggest checking disk space when it runs into the error that I ran into)

[benhoyt]

@tonyandrewmeyer Just fixed the "install tox" section of https://juju.is/docs/sdk/set-up-your-development-environment. However, it seems that the tutorial installs tox but doesn't actually use it! So we should add a section in that tutorial to create and run 1) unit tests, and 2) integration tests.

So updating issue title to reflect this suggestion.

[tonyandrewmeyer]

I just noticed that the canonical/juju-sdk-tutorial-k8s repo has branches for adding unit tests and integration tests. @beliaev-maksim are you in the process of adding these to the tutorial? I'm happy to write those for you based on the code you've got if you'd rather hand it over, but obviously happy for you to finish them off as well!

[beliaev-maksim]

@tonyandrewmeyer thanks for the proposal, @bschimke95 is already working on it and produced 2 drafts: https://discourse.charmhub.io/t/write-integration-tests-for-your-charm/11991/1 https://discourse.charmhub.io/t/write-integration-tests-for-your-charm/11991/1

although, we are very interested in producing a topic that will extend the tutorial with open-port functionality that was recently added to juju and ops

is that something of your interest to contribute to?

cc: @tmihoc

[tonyandrewmeyer]

Sections for unit, scenario, and integration are now included in the k8s tutorial. Thanks @bschimke95 and @beliaev-maksim!