teemtee
Add a plan of plans feature
This is an extension on plan import. The goal is to be able to reference multiple plans with just one link (fmf id).
Imagine following situation:
an SST-A repo:
/plans/Tier1:
tag:
- tier
discover:
- how: fmf
filter: tag:tier1
- how: fmf
url: https://remote.url
filter: tag:tier1
execute:
- how: tmt
/plans/Tier2:
tag:
- tier
discover:
- how: fmf
filter: tag:tier2
- how: fmf
url: https://remote.url
filter: tag:tier2
execute:
- how: tmt
Now, I want to run all the Tier plans from the SST-A's repo. To include whole sub-tree of plans or filtered plans my plan should look like this:
/SST-A:
plan:
include:
url: https://sstA.repo.url
name: /plans
filter: tag:tier
It could be easily internally converted to the known problem:
/SST-A:
/plans/Tier1:
import:
url: https://sstA.repo.url
name: /plans/Tier1
/plans/Tier2:
import:
url: https://sstA.repo.url
name: /plans/Tier2
Which in turn would effectively be:
/SST-A:
/plans/Tier1:
tag:
- tier
discover:
- how: fmf
filter: tag:tier1
- how: fmf
url: https://remote.url
filter: tag:tier1
execute:
- how: tmt
/plans/Tier2:
tag:
- tier
discover:
- how: fmf
filter: tag:tier2
- how: fmf
url: https://remote.url
filter: tag:tier2
execute:
- how: tmt
+ correctly cloned remote repository for the local discovery to work.
"Me too". In https://github.com/cockpit-project/cockpit/pull/19117 I am exploring how to do effective reverse-dependency testing between upstream tests. For example, we'd like to run cockpit-podman's tests in all https://github.com/containers/podman PRs, or cockpit's tests in all systemd or fedora-selinux PRs. It's not possible via packit, and it's a bit awkward with tmt right now as all these projects (selinux, etc.) have to duplicate the plan structure and thus they are hard to change or add.
This seems to be true for both importing plans (as name: is required) and also for the standard discover: url method.
So, some way of "run every plan that you find in url: http:/..." would be very useful. Thanks!
I'm trying to understand the difference between what tmt already has - "take a plan from a repo and replace the current plan with it" - and what's the proposal here. Is it "just" about taking multiple plans from a repo, and sort of replacing the current plan with all of them? Sorting them under the current plan as its children, but effectively doing import of multiple plans instead of one?
@happz That sounds more or less what we need, but more like a "recursive import". E.g. we want to run cockpit-podman's three plans in podman. Currently podman's (and all other consumers, like SELinux has to keep a duplicate of cockpit's plan structure, and kept in sync. This doesn't scale well if the plans change over time, as that change has to be applied to all other projects which want to run these "foreign" plans.
A naive approach of just pointing to a foreign repo and not duplicating the plan structure results in a single plan which runs all of the repo's (intended) plans as tests -- i.e. serially in a single instance.
Hi @sopos and @martinpitt, here is a summary after discussion - add plan of plans.
- A single plan would expand into many plans, and parent-child relationship will be used instead of replacing the local plan.
- For the naming, we would like to concatenate local
(referencing)plan name + remote(referenced)plan name. e.g./plans/local/plans/remote /plans/something - For selecting plans, it is very likely to support to select by a regular expression. By default, all plans would be imported when name is not provided.
- To trigger the new functionality, just omit the name or use a regular expression. And the implementation would be backward compatible with the sing one-one mapping.
- To support mixing local config with remote content, tmt should complain about such a combination:
execute: how: tmt script: echo plan: import: url: here name: (optional) REGEX - To avoid recursion, we would not pay efforts to implement it from the start. We can consider it in the future.
Hi @psss, @happz, @lukaszachy and @thrix, please help to correct the summary above if anything is incorrect, thanks very much!
Hi @psss, @happz, @lukaszachy and @thrix, please help to correct the summary above if anything is incorrect, thanks very much!
+1.
One thing I'm not sure about is how the feature is enabled: not specifying the name is fine, but "use a regular expression" - /foo is also a regular expression, matches /foobar and many other strings, yet it contains no interesting regex-ish characters like dot. It's very hard to tell whether a string is not supposed to be a regular expression...
I think we should add a flag, another key to enable this feature, to make it clear that the user really wants us to import remote plans - even just a single plan! - under the local one. We do have the whole import mapping to play with.
plan:
import:
url: https://src.fedoraproject.org/tests/binutils
# No name set, imports all plans.
import:
# Not a regular expression, but all plans in the repo start with `/plans`.
name: /plans
import:
# Not a regular expression, yields just a single plan - shall we replace the local
# one, or did the user want it to be put under the local plan?
name: /plans/build-gating/common
#
# With a flag telling tmt to "import and slot under the local plan". Let's call it `integrate` for now.
#
import:
url: https://src.fedoraproject.org/tests/binutils
# No name set, imports all plans.
# integrate: true -> nothing changes, remote plans slotted under the local plan.
# integrate: false -> raise an error, tmt is not replacing a single plan with multiple remote ones.
import:
# Not a regular expression, but all plans in the repo start with `/plans`.
name: /plans
# integrate: true -> nothing changes, remote plans slotted under the local plan.
# integrate: false -> raise an error, tmt is not replacing a single plan with multiple remote ones.
import:
# Not a regular expression, yields just a single plan - shall we replace the local
# one, or did the user want it to be put under the local plan?
name: /plans/build-gating/common
# integrate: true -> slot the remote plan under the local one. This would be a new
# behavior when 1:1 import happens, special case of "import and
# slot under" for a single plan.
# integrate: false -> replace the local plan with the remote one. This is the current
# behavior of "import a plan".
This seems to be an important issue for the kernel folks as they would like to import multiple plans from their dedicated plan repository, @bgoncalv can provide some more details. Let's try to implement this in the near future. Proposing for 1.35.
I also have a use case for this feature, and I am volunteering to test it during its development.
Our team still has a use case for this feature, feel free to reach me to better understand our needs.
Same here. Having to copy entire plans between projects for revdeps testing is error prone and not scalable.
As discussed today: we will be removing this issue from a milestone, but do not despair! It is a complex issue, its implementation will not happen in a single milestone anyway. Our plan is to start by finalizing and merging https://github.com/teemtee/tmt/pull/2907 first - ignore the --max bit, the important part of the PR is that it extends existing infrastructure to allow tmt to understand the concept of replacing a single plan with multiple plans. On top of that, we will build the plan-of-plans feature proposed in this issue.
There are still some discussion points left to settle, see https://github.com/teemtee/tmt/issues/1770#issuecomment-1983297612 and https://github.com/teemtee/tmt/issues/1770#issuecomment-2001920770
Time to start hacking... I plan to add new flag to control the "replace or put under" behavior, as I proposed in https://github.com/teemtee/tmt/issues/1770#issuecomment-2001920770
We are removing the discuss label from here, as the path forward is clear, see #3566 for the next steps.
I marked https://github.com/teemtee/tmt/pull/3566 as no longer a draft. Whoever is interested in this feature - @sopos @martinpitt, still? - please, check out docs linked in https://github.com/teemtee/tmt/pull/3566, and if you find some spare time to give a test ride, I'd be very happy to hear back from you.