change test suite to run from a container, or to not require e.g. python-coverage

[cgwalters]

I use rpm-ostree for my host systems (both server and desktop), and do development type things in containers. Currently, I'm trying to run test.sh from a privileged container with the docker.sock bind mounted in. This fails because atomic mount basically assumes it's on the host.

Another approach is to run the tests from the host, but it requires development things like python-coverage which shouldn't be installed on host systems (in this model).

So, I propose we pick one path or the other. The simplest is probably to drop the coverage tooling?

[cgwalters]

Related to this, the tests try to not be destructive but in practice, we write to the host docker, and in the case of system containers, to the host systemd for that matter. I think it'd be better to take a hard cutover and require a VM/vagrant based workflow.

[rhatdan]

Running this in a container is not an option, since SELinux would be disabled in the container, and would not fully test it. Running it in a VM would work, since it is destructive. How difficult would it be to run it in a VM?

[cgwalters]

I've been using a Vagrant based flow, using sshfs for data sharing. Basically I'm doing:

cd ~vagrant/sshfs/src/github/projectatomic/atomic
cd tests/integration
env WORK_DIR=/tmp/foo ATOMIC=/usr/bin/atomic ./test_system_containers.sh

This leads towards switching to an InstalledTests style model. Here, we stop trying to do things like set PYTHONPATH to the sourcedir - the tests always test the installed code.

Cockpit has both make check style unit tests (which run unprivileged and are nondestructive) and their integration tests which use VMs.

IME, vagrant-sshfs is pretty nice at allowing the guest to read data from the host (the source code). Basically the flow is something like:

• hostcontainer$ make (in container on host or container in VM)
• hostcontainer$ make rpm (or equivalent)
• vm$ sudo rpm -Uvh ~vagrant/sshfs/atomic.rpm (This works best with ostree admin unlock)
• vm$ ~vagrant/sshfs/atomic/test.sh