tmt icon indicating copy to clipboard operation
tmt copied to clipboard
verified publisher icon teemtee

[RFE] Define test specific variables as a part of a test listing in discover

Open kkaarreell opened this issue 3 years ago • 3 comments

It would be handy if it would be possible to define test parameters for a test as a part of a test name. Example how it could look like:

discover:
  test:
    - /my/test/foo?PARAM1=bar&PARAM2=baz

Clearly such parameters would have to be urlencoded or something similar. Additionally, somehow it would have to be defined/clear that such a test name+parameters won't be understood as a test name pattern and matched. This could be a bit tricky since test name patters could be regular expressions containing also character ?. ATM I do not have a solution for that.

My use case is that I want to parametrize tests in my plan and cannot use environment definitions because that particular test can be used in the plan multiple times, each time with different parameters. I know I can create virtual (?) tests like /my/test/foo/setup1 where environment definitions can be defined but this won't work in a situation where a) parameters may differ and it won't be practical to create extra virtual test for each option and b) I may not be the owner of that actual test repository.

A real-life example could be a tmt alternative to /distribution/install/brew-build task where users would specify build NVR or buildID to be used during an installation.

kkaarreell avatar Jan 16 '23 17:01 kkaarreell

My 2 cents: I for one find encoding parameters into the test extremely displeasing :( Parsing them out of there, even realizing there's anything to extract first, could be ridden with corner cases. I'd rather see something more structured, making it explicit what the optional variables are, especially when tmt does have an easy-to-discover environment directive for that.

IIUIC, the example is a how: fmf with test: directive for limiting the list of discovered tests. Maybe we could expand the definition of this "filter" of fmf plugin to allow parametrization, e.g.

discover:
  test:
    - name: /my/test/foo
      environment:
        PARAM1: bar
    - /my/other/test/picked/by/name/with/no/extra/touches
    - name: /my/test/foo
      environment:
        PARAM1: baz

AFAICT, how: shell already supports this, the following should work:

discover:
  how: shell
  tests:
    - name: /help/main
      test: tmt --help
      environment:
        PARAM1: bar

I for one would find applying the approach to fmf plugin to be an improvement, making the "tweakability" support of discovered tests closer across different plugins.

how: fmf
test:
  - name: /distribution/install/brew-build
    environment:
      BUILD: foo-1.2-3.el9
  - /run/testsuite

how: shell
tests:
  - name: Install a brew build
    path: /distribution/install/brew-build
    test: ./runtest.sh
    environment:
      BUILD: foo-1.2-3.el9
  - name: Run testsuite
    path: /run/testsuite

Both plugins would support the very same way for defining environment variables for that particular discovered test (or tests if name would be a pattern yielding multiple tests).

happz avatar Jan 16 '23 21:01 happz

That would be definitely better.

kkaarreell avatar Jan 16 '23 22:01 kkaarreell

We could also really use a similar feature.

All our our tests use result:custom reporting, creating one result for each OpenSCAP rule scanned, ie.

/hardening/oscap/<profile>/<rule>
/hardening/oscap/ospp/firewalld_ssh_port_enabled

This is mostly fine for tests that finish in some reasonable amount of time, so they don't really need to be parametrized, but it means we have to hardcode at least <profile> in FMF metadata, and duplicate it many times for all tests that run on all profiles, see ie.

  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/oscap/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/oscap/with-gui.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/anaconda/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/anaconda/with-gui.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/ansible/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/ansible/with-gui.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/host-os/oscap/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/host-os/ansible/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/image-builder/main.fmf
  • https://github.com/RHSecurityCompliance/contest/blob/main/hardening/image-builder/with-gui.fmf

For these specifically, it would be great if we could have ie. just one FMF-defined /hardening/oscap test, which would programmatically list all profiles on the system, run them, and report /<profile>/<rule> via result:custom.

But, for that to be usable, we would need support for running the /hardening/oscap test with profile or profile+rule name in the test name itself, so a user could run tmt run ... tests -n /hardening/oscap/ospp to test the ospp profile, or /hardening/oscap/ospp/some_rule to test some_rule specifically.

Passing env-variable-looking parameters like OP suggested doesn't seem like a nice approach here.

Similarly, another test which could really use it is /per-rule, which runs upstream unit tests for each rule, and reports results like

/per-rule/oscap/some_rule_name
/per-rule/oscap/another_rule_name
/per-rule/ansible/yet_another_rule

and the problem with /per-rule is that it doesn't finish in reasonable amount of time, it may take almost a full day to test all rules.
In Automilos (bug verification), we might want to check just one or two rules that were impacted by the bug fix, so we need an env var parameter to override the "run all rules" logic.

RULE="some_rule another_rule"

which is fine, but it would definitely be nicer if we could specify it in the test name

/per-rule/[^/]+/test_this_rule

We have more tests like this, probably also similar to the OP example, where we would specify a matrix to be passed to the test, ie.

/scanning/some_fmf_test/uefi,virtio-blk,i440fx
/scanning/some_fmf_test/uefi,virtio-scsi,i440fx
/scanning/some_fmf_test/bios,virtio-scsi,q35

(or alternatively)

/scanning/some_fmf_test/bios/virtio-scsi/q35

(or, as parsed by the test)

/scanning/some_fmf_test/boot=bios,net=virtio-scsi,platform=q35

either specified in a TMT plan, or by the user via discover -t or tests -n.

(TMT would obviously process only the /scanning/some_fmf_test part and give the rest to the test.)

Implementation-wise, this would be best done (IMHO) via CLI arguments to the test executable, appended after test:, so that /scanning/some_fmf_test receives one argument with the right side of / after its name.

test: ./runtest.sh

would be run as (argv):

['./runtest.sh', 'bios,virtio-scsi,q35']
['./runtest.sh', 'bios/virtio-scsi/q35']   # example with slashes

TMT would have to include extra logic to try the given discover -t or tests -n (discover:test: in a plan) arguments against the full discovered list of tests, as it presumably does now, but like this:

  • try to match the user-specified expression against all tests
    • if it succeeds, use current behavior
    • if that fails, do str.rpartition('/') on it and try again with the left part, keeping the right part in a temp variable
      • if it fails again, try rpartition again, prepending the next right-cutoff to the temp variable
      • if a match against a test name is eventually found, use the matched test / tests, passing the collected temp variable as $1

ie. parsing:

discover:
    how: fmf
    test:
        - /hardening/[^/]+/ospp/some_rule
  1. collect real fmf-defined test names
    • /hardening/oscap
    • /hardening/ansible
    • /hardening/anaconda
  2. try matching /hardening/[^/]+/ospp/some_rule against each discovered test, as a regexp
  3. try '/hardening/[^/]+/ospp/some_rule'.rpartition('/'), take /hardening/[^/]+/ospp as new test match, some_rule as temp var (python list, ['some_rule'])
  4. try matching /hardening/[^/]+/ospp against each discovered test
  5. try '/hardening/[^/]+/ospp'.rpartition('/'), take /hardening/[^/]+ as new test match, prepend ospp to temp var, ['ospp', 'some_rule']
  6. found match of /hardening/[^/]+ on /hardening/oscap, schedule it to be run with arg: '/'.join(['ospp', 'some_rule'])
  7. found match of /hardening/[^/]+ on /hardening/ansible, schedule it to be run with arg: '/'.join(['ospp', 'some_rule'])
  8. found match of /hardening/[^/]+ on /hardening/anaconda, schedule it to be run with arg: '/'.join(['ospp', 'some_rule'])

I admit, maybe not the cleanest way given that TMT already treats user-specified test names as regexes, but it seems TMT already understands discover -t or tests -n being given the same test multiple times, so it already must have some logic for iterating over them, rather than just filtering discovered FMF tests.

There are more ways this could go on, ie.

/hardening/oscap/ospp/some_rule
/hardening/oscap/stig/some_rule

could be "merged" into one test run, passed as $1 and $2 respectively,

['./test.sh', 'ospp/some_rule', 'stig/some_rule']

as I do in PTEF: running-with-arguments and argument-merging, but that's probably overkill for now.

comps avatar May 10 '24 11:05 comps