Support "AND" logic to tags for the tests runner

[alexeyhimself]

Is your feature request related to a problem? Please describe. We use different kinds of tags to formulate a scope for the tests run. We have different kinds of tests: post deployment verification (pdv), functional, data quality evaluation (dqe) tests, unit tests and so on. We have different kinds of releases (relatively defined: current, next; explicitly defined: v63_0, v63_1, v64_0) as well. And we'd wanted to run test cases among these categories.

For example, we'd wanted to run all "p0 tests in v64_0 release". We have tests tagged with priorities: "p0", "p1", "p2", "p3". We have tests marked with category: "pdv", "dqe", "functional", etc. We have special tags like: "long_running", "not_for_production", etc. We have release tags: "v63_0", "v64_0", etc. And we tag our tests, for example, as follows:

--%tag(v64_0, pdv, not_for_production)
--%tag(current, pdv)
--%tag(current, p0, functional)
--%tag(current, p0, performance)
--%tag(current, p1, functional)
--%tag(v63_1, dqe, long_running)

When we want to run all "p0" tests in "v64_0" release, we have no way to do it now. We either run all "p0" tests or all "v64_0" tests, or both - due to "OR" logic for the tests run. We want both "OR" and "AND" logic as well - to form different kinds of tests scope.

Describe the solution you'd like We'd wanted you support this kind of separators to apply different kinds of logic to the tests runner:

select * from table(ut.run(a_tags=>'p0||p1')); -- "OR" logic
select * from table(ut.run(a_tags=>'p0&&v64_0')); -- "AND" logic
select * from table(ut.run(a_tags=>'(p0||p1)&&v64_0')); -- mixed logic with priority defined by ()

Describe alternatives you've considered Alternative to this - is to use self-written tests installer to cleanup all test procedures before tests run and install only required tests and run all installed tests. I.e. alternative - is to delegate this scope defining function to the self-written installer. This is not useful and not nice. And this installation-deinstallation packages activity produces db system logs that we don't want. We'd better installed our tests once and then run subsets of them on demand.

Additional context No

[jgebal]

Hi @alexeyhimself

Thanks for raising this request.

This kind of logic will require quite a bit of parsing and error handling. The logic for building included items (suites/parent suites/tests) would also become quite complex.

Though the request you're making is absolutely valid, I don't see an easy way of implementing it at the moment. Also, we currently suport comma-separated lists of tags to include/exclude (as or logic) we would need to maintain backward-compatibility.

[jgebal]

@alexeyhimself

In our current implementation we use the following logic:

1. Split the tags into wo lists (include/exclude) based on the - prefix in command.
2. Filter suite items to be executed by using IN / NOT IN logic

The type of filtering you propose would mean building a complex SQL condition dynamically. So instead of having a list of items that should be used as filters we would have a list of expressions.

Each of expressions would need to be evaluated separately. Only whole expression could be used for inclusion or exclusion.

It is an interesting challenge.

Do you have any idea on how would you apply such filter criteria to a SQL table or a nested table/varray?