glusterfs
glusterfs copied to clipboard
gluster
tests: new testing tool
A new tool gftest has been created to automate and simplify the preparation of an environment to run tests. It can also process the full set of regression tests in parallel to significantly reduce the total execution time.
It's written in python and requires these components:
- libvirt
- qemu-kvm
- gnupg
- python-click
All other modules should already be present in a default installation.
Using this tool is quite simple:
- Create the VM
gftest create [OPTIONS] <name>
Options:
--template <name> Template to use (default: centos7)
--port <num> Port to access the VM (default: 2222)
--key <path> Private key for SSH (default: ~/.ssh/id_rsa)
--cpus <num> Number of virtual cores of the VM
--memory <num> Memory assigned to the VM in GiB
--disk <num> Disk space for the VM in GiB
This creates a KVM VM named <name> that can be accessed through the
specified port on the local host (i.e. ssh -p <port> [email protected])
using the specified private key. All other option will be taken from
the template if not present.
This process also creates a container inside the VM with all the required packages and configurations to build Gluster and run the tests.
- Build the code
gftest build [OPTIONS] <name>
Options:
--commit <id> SHA/name of the commit to compile
This copies the current git repo to the VM and compiles the specified commit in a container image.
- Start the workers
gftest spawn [OPTIONS] <name>
Options:
--workers <num> Number of workers to create
--space <num> Space allocated for running tests in GiB
This starts the specified number of container instances of the compiled Gluster image.
- Run the tests
gftest run <name> <output dir>
This runs the full set of tests in parallel in all available containers and puts the results into the <output dir>.
It also collects statistics for each test and CPU and memory state during the full run.
- Access the VM or container for manual testing/debugging
gftest sh <name> [<idx>]
Without idx it opens a shell session to the VM. With an index it opens
a shell session to the idx-th testing container.
There are some additional commands:
-
gftest kill <name>Stops and destroys all testing containers. -
gftest shutdown <name>Gracefully shuts down the VM. -
gftest poweroff <name>Forcibly stops the VM. -
gftest poweron <name>Starts the VM.
Updates: https://github.com/gluster/glusterfs/issues/3469 Change-Id: I169877a4c5197d001bf2822ad7116559f7d78754 Signed-off-by: Xavi Hernandez <[email protected]>
This PR won't work without all the other PRs related to #3469.
Example usage:
# gftest create demo Validating checksum's signature... Verifying VM image's checksum... Creating VM disk... Installing VM... Restarting VM... Linux demo.glusterfs.local 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Creating builder container... STEP 1: FROM centos:7.9.2009 Getting image source signatures Copying blob 2d473b07cdd5 done Copying config eeb6ee3f44 done Writing manifest to image destination Storing signatures STEP 2: ENV container docker [...] STEP 28: COMMIT glusterfs/builder Getting image source signatures Copying blob 174f56854903 skipped: already exists Copying blob 1f271ebf5eb8 done Copying config 50dfe82c41 done Writing manifest to image destination Storing signatures 50dfe82c41262a92d7a81d97166c915a1f3bfab78287aaeee134c7b5f7c83af6 50dfe82c41262a92d7a81d97166c915a1f3bfab78287aaeee134c7b5f7c83af6 # gftest build --commit gftest-testing demo Cloning into '/root/glusterfs'... STEP 1: FROM glusterfs/builder [...] STEP 10: COMMIT glusterfs/testing Getting image source signatures Copying blob 174f56854903 skipped: already exists Copying blob 3377ad3ed6a3 skipped: already exists Copying blob d45403686810 done Copying config 8d173cf3d0 done Writing manifest to image destination Storing signatures 8d173cf3d0439606d1708606172ffa7223a616f0ede8af30c8a56583919d32db 8d173cf3d0439606d1708606172ffa7223a616f0ede8af30c8a56583919d32db # gftest spawn demo Physical volume "/dev/loop0" successfully created. Volume group "vg_demo" successfully created Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data. Logical volume "tp" created. Logical volume "lv_demo_00" created. b44e331979deeb24139f294ccaa1d73562c9ba55db3cb0c63c3efdde6273388e [...] # gftest run demo /tmp/run 800 test(s) found 1/800 0.4 [demo01] 21.7 1 PASSED tests/00-geo-rep/bug-1708603.t 2/800 0.9 [demo02] 53.0 1 PASSED tests/00-geo-rep/bug-1600145.t [...] 798/800 52.0 [demo01] 25.0 1 PASSED tests/line-coverage/shard-coverage.t 799/800 53.1 [demo00] 157.7 1 PASSED tests/features/worm.t 800/800 53.5 [demo04] 142.7 1 PASSED tests/line-coverage/errorgen-coverage.t
In my laptop (i7 7600U 2.8GHz, 2 cores, 4 threads) I've been able to run all 800 tests in little more than 50 minutes. In a more powerful machine it takes less than 15 minutes.
The tool also uses information from previous executions (if the passed directory is the same) to order the tests based on their average execution time. Slower tests are run first to maximize parallelization at the end. This can save few additional minutes.
Finally, to make it easy to access the VM or any of the containers, this command can be used:
gftest sh <name>
This opens a shell to the VM.
gftest sh <name> <n>
This opens a shell to the nth worker container.
Tool itself is good to get in. Only thing is, in future, should we separate it out as a different project / repository?
The idea of the tool is to provide a replacement for the regression test execution suite so that it can be done faster, and since the regression scripts are already in this repository, I didn't thought about creating it externally, but surely it could be considered if its features are expanded further.
I did a quick try on a machine with similar specs as the builders used for CI and it took less than 1.5 hours (if I remember correctly), so I think it's something interesting to investigate.
If I'll have time, I'll send a separate PR for some improvements to the libvirt VM defintion. iothreads, writeback cache, host-passthrough CPU, bootmenu removal (5 secs on boot), remove unused devices can all improve execution a bit.
