CI testing for VM/CNV

[dry923]

trafficstars

Spawning this issue from https://github.com/cloud-bulldozer/benchmark-wrapper/pull/261 which was caused because we did not test on anything outside of openshift (ie VMs, etc).

We currently do not have a method of CI testing CNV/VM's for benchmark-wrapper. As we continue to move forward this is something we should look into as it will likely become more prevelant.

cc: @sarahbx @jtaleric @amitsagtani97

[bengland2]

@dry923 Just FYI, if you add CNV to the CI environment, we can run fio inside CNV VMs, that's already supported (except for issue 527 which I haven't had time to work on, but it has an easy fix described there).

[jtaleric]

Hey @bengland2 The hope would be we increase the coverage of SNAFU to cover, BM, VM, Container (Podman/docker), OCP...

[sarahbx]

+1 for VM/CNV testing, but we should probably look at moving away from python setup.py develop for CI builds and normal usage. That was the main difference between the two in #261.

[amitsagtani97]

Issue #261 was caused because of missing init file in the log_wrapper directory. It would still fail on outside of openshift (ie VMs, etc) without that file. benchmark-wrapper is installed as a python package and directories without init file are not included in the package.

Benchmark-wrapper CI uses benchmark-operator, the test for log_wrapper were not merged at that moment, else our CI would have found this issue.

@sarahbx Another way to install benchmark-wrapper is pip install git+https://github.com/cloud-bulldozer/benchmark-wrapper.git but I guess that is similar to git clone and python setup.py develop

[sarahbx]

Hi @amitsagtani97 :) For VMs in benchmark-operator, we are using pip install git+https://... which behaves as python setup.py install. When running the pod versions of the same tests, they were successful. Looks like pip install -e is being used for the wrapper installs, which behaves the same as python setup.py develop.

https://github.com/cloud-bulldozer/benchmark-wrapper/blob/19c9e3bd3ad3f5bafb90648c31ecac9075c92643/snafu/uperf_wrapper/Dockerfile#L11

[jtaleric]

@learnitall I wonder how your staged CI would mitigate the need for this?