content
content copied to clipboard
Templating of test scenarios in the test suite is messy
How templated tests are handled:
- Templated tests are picked up at the moment of rule collection.
- Duplicate scenarios are filtered out by default (if tests of multiple rules of the same template are requested)
- Tests are aggregated by the rule ID (including templated scenarios), and dispatched to the remote machine.
- Tests are executed - previous events ensure that requested scenarios are available on the remote environment.
Issues with this:
- Templated tests are identified twice - once so scenarios can be collected, and for the second time so that they can get transferred to the testing machine. This increases maintenance costs. A cleaner approach would be to make a list once and then just pass it on in a read-only mode.
- The function rule_should_be_tested got really complex, it is no longer clear what are the criteria that result in "pass".
- The preference of rule scenarios over templated ones is achieved by performing file copy in a certain order, the intention to prefer something doesn't have any imprint in the source code.
Some of the items will be improved by https://github.com/ComplianceAsCode/content/pull/8751
The templated tests are no longer collected twice since https://github.com/ComplianceAsCode/content/pull/8883