As a tester, I need to have a way to select and insert 'workflow' level metadata when scheduling a test job with tmt.

For example, there is a workflow where AVC check is required. As of now, that can be achieved via main.fmf in components' test root or via test plan. The problem is, that this piece of metadata certainly is not component-specific so the former approach is inappropriate (and I consider it harmful) and the latter approach is way too fragmented.

Simply put, when we want to introduce a new check in the workflow (or remove one from it), it should be doable via single action (as opposed to commiting to hundreds of test roots).

For me as a tester, it would be very useful to have a way to say '"I'm testing RHEL, I want AVC check and ABRT check and this or that setting" or "I'm testing RHEL in FIPS mode, I want FIPS enabled and AVC check" or "I'm testing distro abc and I don't want any checks but I want this or that setting."

The 'workflow' metadata or profile (however you want to call it) shall be selectable via cmdline (--profile/--workflow RHEL).

Questions to consider:

metadata precedence: if there's a conflict between metadata from test, plan and commandline - what should prevail?

test metadata are the most specific and workflow metadata are the least
but cmdline expresses the user's explicit will and having it modified in the pipeline by something else leads to a great confision (where the hell did that come from?)

https://github.com/teemtee/tmt/issues/3611

Feb 18 '25 12:02 martinky82

I am probably facing similar problem. I am mantaining tests and plans in multiple repositories but there is some common setup I would like to maintain in a single copy. ATM I have multiple copies of main.fmf but if I would be able to "include" main.fmf from a remote path (not necessarily whole tree, just single main.fmf file would be enough) my life would be easier.

Feb 26 '25 13:02 kkaarreell

I realized that newa comfiguration (enabling tmt repotportal plugin and so on) should be for most of the components also defined in the workflow/profile metadata. Indeed, in main.fmf should be only component-specific metadata, not things that are be the same for most or even all of the components when testing a specific test scenario such as errata testing , RHEL CI and so on (which needs to be copied over and over under current approach). Let's have sane, standard workflows defined centrally and let users tweak it by main.fmf if they need to.

Mar 10 '25 12:03 martinky82

Summary from the hacking session discussion:

Limit this to the check scenario only?
- There are similar use cases, e.g. enable reportportal report for all plans in the CI pipeline
- Decision: no, we need some more generic solution which would allow modifying the tmt command line
What are examples of the default setup?
- enable the avc check for all tests (unless individual test specifies otherwise)
- perform extra preparation step on the guest (e.g. prepare --insert --how feature --epel enabled)
- report test results to the common location (e.g. report --insert --how reportportal)
What about clash between default setup and test config?
- Keep in mind: Some tests need to disable or modify the check
- Example: Test intentionally triggers an avc failure
  - profile adds avc check: result: respect
  - test defines avc check: result: xfail
- Example: Test wants to completely disable the specific check
  - profile adds avc check: result: respect
  - test defines avc check: enabled: false
Workflows
- There are several workflows where the defaults would be applied
- Defaults can be different across workflows, thus each workflow would store its config
- Examples: gating, newa, compose qualification, component update testing
- Keep in mind: We want to make easy for users to reproduce problems, thus the configs need to be easily accessible so that tmt run un user laptop works the same way
Way how to deliver
- /etc/tmt … global or user config files?
- TMT_CHECK_DEFAULT … environment variables not powerful enough?
- command line snippets … seem to be the most flexible?
Where should the modification happen?
- fmf is not the right place, the merging functionality not powerful enough
- We need to handle defaults, data normalization and combining profile with test, thus tmt code will have to handle this
Will it be possible to combine several profiles together?
- This could be useful, we could have
  - mandatory defaults for all products (e.g. avc check)
  - workflow specific adjustments (e.g. disable epel)
Next steps: Finalize the scope
- Interested: @psss, @thrix, @kkaarreell, @lukaszachy, @happz, @martinky82

Mar 21 '25 09:03 psss

It is important to catch any possible avc denials as soon as possible. This has been escalated as it can present a singficant problem to customers and support. Proposing high priority. Possible / temporary workaround is that each component enables the avc check but that would require changes in many repositories.

Mar 21 '25 09:03 psss

It would be maybe beneficial not to disable AVC check entirely when AVC is expected, but make it informational only, ie. the logs will be there but AVC fail would not trigger test failure.

Mar 25 '25 10:03 martinky82

As for the AVC: because of historical beah/restraint usage affected tests are likely to set AVC_ERROR=+no_avc_check environment variable. Maybe we could honor it as well.

Apr 03 '25 07:04 lukaszachy

As for the AVC: because of historical beah/restraint usage affected tests are likely to set AVC_ERROR=+no_avc_check environment variable. Maybe we could honor it as well.

Sounds ok to me. I'd recommend to enable the backward-compatible behaviour only when the new restraint-compatible option is enabled. To allow both fluent transition of existing tests and future cleanup of the deprecated code.

Apr 07 '25 13:04 psss

Summary from the sync meeting:

There are several dimensions of the problem:

metadata level — test config (e.g. avc check) ⨯ plan config (e.g. insert an extra prepare step)
component scope — pipeline defines a default (from up) ⨯ a component subscribes a common config (from down)
delivery method — command line option (e.g. prepare --insert …) ⨯ stored fmf files (essential for larger stuff?)
profile merging — fmf metadata (tree grafting or dict merge) ⨯ tmt dict merge (with defaults & normalization)

For the avc check MVP it was agreed to limit the scope in the following way:

metadata level — test config
component scope — pipeline defines a default (from up)
delivery method — fmf files enabled through an option
profile merging — most probably in tmt to handle defaults/normalization

A couple thoughts related to the delivery method:

just command line adjustments seem like a workaround, syntax can change
specification should be strict and stay the same (should be safer)
--profile profile.fmf (the content itself is written in fmf)
config in git would be more easy to review & track, maintain, traceability (how exactly the test was run)

Priority of the source needs to be clearly defined and documented:

What is a more specific pipeline or component?
From a point of view, pipeline is more specific (e.g. exact compose)
Here one specific test overrides

Example command line brainstorm:

tmt run --profile avc.fmf

Example profile file brainstorm:

test-profile:
    check+:
      - how: avc
        result: info

plan-profile:
    prepare:
      - how: ansible
        playbook: rhel-ci.yaml
        order: <some-agreed-order>
      - how: feature
        epel: disabled
        order: <some-agreed-order>

Apr 23 '25 08:04 psss

Example command line brainstorm:
tmt run --profile avc.fmf
Example profile file brainstorm:

A slightly modified example:

test-profile:
    check+:
      - how: avc

plan-profile:
    prepare:
      - how: ansible
        playbook: rhel-ci.yaml
        order: <some-agreed-order>
      - how: feature
        epel: disabled
        order: <some-agreed-order>

A couple of questions:

what is supposed to happen in the case of check+ and prepare? Naively, I would expect the profile to add avc check to all tests, and set prepare step of all plans to just the ansible and feature phases listed above. Is that a correct expectation?
do we wish to support removal of step phases/test checks?
how should the stricter "no" from the test/plan look like when test/plan authors wish to prevent these additions?
- the possibility to "neutralize" a particular check was discussed, how would that be expressed in test metadata, e.g. in a test that absolutely does not wish to fail on AVC denials? Using respect: info rather than removing the check completely seemed to be preferred, how would the test's wish be matched against the profile's forced check?
- Imagine a plan that absolutely requires EPEL to be enabled. What should happen?
how about a profile also adding some context dimensions? I can imagine a plan that requires EPEL might be better to disable itself when facing the epel: disabled phase mandated by a profile - if the profile could set "something" for the adjust to consume, such a reaction could be possible. It may also work for the "but we need to silence AVC check" tests.

Apr 24 '25 07:04 happz

I got many answers on Friday, many answers in fact, and gave it some thinking being stricken by illness & unable to do much else. I tried to follow the approach of "instructions" or "directives" provided by the workflow maintainers which would tmt decode and follow to remodel any given test. Some interesting realizations: "update if missing" we know from plan-level changes is harder to model, as by this point, all values are already materialized; various "add new contact", "remove tag" and "add foo if it isn't there already" requirements lead to a complicated set of controls hen combined with various data types - lists of things, mappings of things, namely, and "things" are not necessarily strings or integers, but also checks and links, i.e. mappings.

All in all, a lot of headache when I was attempting to model some kind of DSL that would let us "add AVC check but only if the test does not have it yet", with some neat addressing of keys and their subkeys ("a check with how equal avc"). Tricky keys, suffixes and prefixes and how they would apply to a list with items and a list without items, and so on. Eventually, I gave up after realizing we would be reinventing the wheel (many of them...), and I went for a simple yet powerful answer to everything: be explicit, strict & template the hell out of it, https://github.com/teemtee/tmt/pull/3702 - added for test export only, with export --how=yaml it's easy to play with.

The idea is simple: every directive contains one or more test metadata keys we know and love, their values would be templates that, when rendered, would be parsed as if they would have been provided by an fmf file, then normalized as any other fmf input, and replace whatever was the original value of the given key. Give them access to the original value (VALUE) and the test (TEST), and it's done. A couple of examples:

every test would suddenly run with GCC Toolset software collection enabled:
```
test-profile:
  - test: |
      scl enable gcc-toolset-15 {{ VALUE }}
```

ok, maybe not every test, maybe just those owned by one the many SSTs:

test-profile:
  - test: |
      {% if "[email protected]" in TEST.contact %}
      scl enable gcc-toolset-15 {{ VALUE }}
      {% else %}
      {{ VALUE }}
      {% endif %}

"we want to add AVC check to every test"

test-profile:
  - check: |
      - how: avc
        result: respect

      {# Make sure to include checks already picked by the test #}
          {% for check in VALUE %}
      - {{ check }}
          {% endfor %}
      {% endif %}

"but... but... but what about tests that do produce AVC denials? they would start failing, people would not like us anymore!"

test-profile:
  - check: |
      {#
        If no check has been defined for this test, inject the default
        AVC check. The current value would be false-ish, as an empty list.
      #}
      {% if 'avc' not in VALUE | map(attribute='how') %}
      - how: avc
        result: respect
      {% endif %}

      {# Make sure to include checks already picked by the test #}
      {% for check in VALUE %}
      - {{ check }}
      {% endfor %}

A couple of pros:

known tooling
no magic keys, suffixes, characters, behavior: key: template -> render the template -> treat it as fmf, normalize, replace the original
update, update-if-missing, remove, all possible with if/else flow controls and access to the original value

easy to add syntax sugar to simplify common operations, like "include the original value", would make the template more readable. For example, has_check test and ORIGINAL string holding the original value as YAML snippet:

test-profile:
  - check: |
      {#
        If no check has been defined for this test, inject the default
        AVC check. The current value would be false-ish, as an empty list.
      #}
      {% if not TEST | has_check('avc') %}
      - how: avc
        result: respect
      {% endif %}

      {# Make sure to include checks already picked by the test #}
      {{ ORIGINAL }}
      {% endif %}

I like it, especially after spending significant time on that DSL idea, e.g.

test-profile:
  - action: set-default
    match:
      type: dict
      parent: check
      keys:
        how: avc
    values:
      result: respect

All of those keys would require some kind of code on our side, find subkeys of a key value matching the requirements, a code to modify an object, be it a test or one of its structured subkeys, and so on. Have a template, original value, the test, same parsing, same normalization. Much cleaner. Let me know what do you think.

Apr 28 '25 19:04 happz

Templating-Based Solution – Initial Observations

Note: This is a quick brain dump from tonight. I may update the comment tomorrow.

General Impressions

The proposed solution based on templating looks flexible enough to handle the complexity.
It reminds me of some great work designed by Milos, which served us well in BaseOS CI over the years for managing complex metadata.
The fact that templating is applied after metadata materializes is excellent. This approach would:
- Allow easy generation of diffs for users, showing how each test or plan would be modified by a profile.
- Be straightforward to test: users could run:
```
tmt test export --profile rhel-ci
```
  to see how the test metadata would change when the profile is applied.
- Enable clear comparisons via a --diff flag:
```
tmt test export --profile rhel-ci --diff
```
  to highlight what would change if the test ran with the rhel-ci profile.

Design Considerations for Templating

Here are some things I’d like to see designed in from the beginning:

Documentation:
All templates should come with documentation explaining why the change was introduced—ideally referencing a GitHub Issue, Jira ticket, or other traceable reference showing who proposed and approved the change.
Auditability:
Users should have an easy way to investigate changes made by a profile in past runs. The diff and the applied profile should be included in tmt run metadata to support post-mortem analysis.
Discoverability:
It should be obvious that a test was executed with a specific profile. The applied template should be clearly visible, ideally with a link to profile documentation.
Validation:
After templating is applied, tmt should re-validate the modified metadata. If it’s invalid, the run should be refused.
Consistency:
We should enforce strict design patterns and aim to standardize common use cases. For example:
- When using a testing farm environment collection, always add it in a consistent order.

Key Decisions Before First Release

Some choices should be made before we release the first version:

Multiple Templating on the Same Field:
Should we allow multiple templates to modify the same field (e.g., templating check twice in one profile)?
Combining Multiple Profiles:
Should we support combining profiles? For example, users might want to apply the rhel-ci profile and then modify it with another like rhel-ci-image-mode.
Alternatively, we might prefer multiple flavors of a single profile:
- rhel-ci (base)
- rhel-ci-image-mode
- rhel-ci-package-mode
- rhel-ci-fusa (rhel-ci profile modified for functional safety requirements)
  These could all be maintained in one place.
Expose Profile Info to Tests:
Should tests be able to change behavior based on the applied profile?
If so, we could standardize this by requiring each profile to set an environment variable:
```
TMT_PROFILE=rhel-ci
```

Final Thoughts

We should take the time to clearly define use cases and create example scenarios to validate the idea early on.

May 15 '25 00:05 thrix

A few quick comments:

The fact that templating is applied after metadata materializes is excellent. This approach would:

Allow easy generation of diffs for users, showing how each test or plan would be modified by a profile.

Yes, --diff can be implemented, I believe easily, to display what fields would change and how. Together with logging of changes, where "diff" might be a higher verbosity output, it would indeed provide nice picture of changes applied by a tmt run profile to each object.

Documentation: All templates should come with documentation explaining why the change was introduced—ideally referencing a GitHub Issue, Jira ticket, or other traceable reference showing who proposed and approved the change.

Yes, that should be the norm in all our configuration endeavours.

Auditability: Users should have an easy way to investigate changes made by a profile in past runs. The diff and the applied profile should be included in tmt run metadata to support post-mortem analysis.

This is a long-term pain point with adjust. There should be a "changelog" of some kind, structured or not, reporting what changes fmf and tmt take before running things. There is a logging of adjust, but that is very verbose and on-demand. This would be an opportunity to do something more in this area.

Discoverability: It should be obvious that a test was executed with a specific profile. The applied template should be clearly visible, ideally with a link to profile documentation.

tmt can propagate the tmt run profile name into results, and plugins/TF can visualise the field as necessary.

Validation: After templating is applied, tmt should re-validate the modified metadata. If it’s invalid, the run should be refused.

It does re-validate the data, it treats them as if they come from fmf, normalization is applied.

Key Decisions Before First Release

Some choices should be made before we release the first version:

Multiple Templating on the Same Field: Should we allow multiple templates to modify the same field (e.g., templating check twice in one profile)?

I believe we should. In your example, should there be the desire to add not just avc check, but also coredump check, I would expect these to be implemented by two distinct rules in the run profile, each with its own respective docs, reasoning, and links.

Combining Multiple Profiles: Should we support combining profiles? For example, users might want to apply the rhel-ci profile and then modify it with another like rhel-ci-image-mode.

AFAIK, for now we agreed to deal with one tmt run profile only. To the proposal and the demo implementation, applying multiple tmt run profiles would be trivial, it would be just a stream of instructions.

Expose Profile Info to Tests: Should tests be able to change behavior based on the applied profile? If so, we could standardize this by requiring each profile to set an environment variable: TMT_PROFILE=rhel-ci

Just as tmt can expose the tmt run profile name in results, it can also add the variable to all tests' environments. No need to rely on profiles to do that for us.

Final Thoughts

We should take the time to clearly define use cases and create example scenarios to validate the idea early on.

I tried to demonstrate some of the use cases that were mentioned during discussions, and yes, I would like to see more and try to model them. Including plan-level modifications we agreed would be out of scope for now, it would be good to realize this proposal would never work for plans now rather than later...

May 15 '25 08:05 happz

A few quick comments:

The fact that templating is applied after metadata materializes is excellent. This approach would:

Allow easy generation of diffs for users, showing how each test or plan would be modified by a profile.

Yes, --diff can be implemented, I believe easily, to display what fields would change and how. Together with logging of changes, where "diff" might be a higher verbosity output, it would indeed provide nice picture of changes applied by a tmt run profile to each object.

Ack, thanks you!

Documentation: All templates should come with documentation explaining why the change was introduced—ideally referencing a GitHub Issue, Jira ticket, or other traceable reference showing who proposed and approved the change.

Yes, that should be the norm in all our configuration endeavours.

Agreed

Auditability: Users should have an easy way to investigate changes made by a profile in past runs. The diff and the applied profile should be included in tmt run metadata to support post-mortem analysis.

This is a long-term pain point with adjust. There should be a "changelog" of some kind, structured or not, reporting what changes fmf and tmt take before running things. There is a logging of adjust, but that is very verbose and on-demand. This would be an opportunity to do something more in this area.

Agreed, similar problem we have in adjust, I conclude.

Discoverability: It should be obvious that a test was executed with a specific profile. The applied template should be clearly visible, ideally with a link to profile documentation.

tmt can propagate the tmt run profile name into results, and plugins/TF can visualise the field as necessary.

Yes, exactly, it would be clear what was applied.

Validation: After templating is applied, tmt should re-validate the modified metadata. If it’s invalid, the run should be refused.

It does re-validate the data, it treats them as if they come from fmf, normalization is applied.

Awesome, thanks for confirming

Key Decisions Before First Release

Some choices should be made before we release the first version:

Multiple Templating on the Same Field: Should we allow multiple templates to modify the same field (e.g., templating check twice in one profile)?

I believe we should. In your example, should there be the desire to add not just avc check, but also coredump check, I would expect these to be implemented by two distinct rules in the run profile, each with its own respective docs, reasoning, and links.

I was more thinking about possible issues, when the same fields are edited, thus we will need to define a way how the templates will be applied, in what order. I would be not even against go with simple 10-avc.fmf, 20-avc-override.fmf to simply solve the problem.

Combining Multiple Profiles: Should we support combining profiles? For example, users might want to apply the rhel-ci profile and then modify it with another like rhel-ci-image-mode.

AFAIK, for now we agreed to deal with one tmt run profile only. To the proposal and the demo implementation, applying multiple tmt run profiles would be trivial, it would be just a stream of instructions.

Ack, just saying, rather than combining, and possible side effects of that, I would encourage people to contribute to the profile they want to extend for their use case. Introduce some kind of flavors to directly influence the templates via some variables, etc. That way we do not need to deal with this crap.

Expose Profile Info to Tests: Should tests be able to change behavior based on the applied profile? If so, we could standardize this by requiring each profile to set an environment variable: TMT_PROFILE=rhel-ci

Just as tmt can expose the tmt run profile name in results, it can also add the variable to all tests' environments. No need to rely on profiles to do that for us.

Agreed.

Final Thoughts

We should take the time to clearly define use cases and create example scenarios to validate the idea early on.

I tried to demonstrate some of the use cases that were mentioned during discussions, and yes, I would like to see more and try to model them. Including plan-level modifications we agreed would be out of scope for now, it would be good to realize this proposal would never work for plans now rather than later...

I will try to provide some examples for the plans and guest setup.

May 15 '25 11:05 thrix

@happz, thanks a lot for preparing the draft! Very nice to have already the tmt test export functionality draft available for experimenting!

At the first sight my impression was that this would be a bit complicated approach, quite hard to understand. After playing a bit more with the thought I really appreciate the flexibility. Seems it could be a powerful way to perform various metadata modifications. And I agree that despite a bit verbose/longer syntax, it would be better then introducing some kind of complex DSL or reinventing wheels.

Simplicity

I like the mention of the possible syntax sugar extensions which could make the code more readable / shorter for the most common use cases. That would definitely help.

Thinking about the ORIGINAL example: I guess there might be also some pitfalls, like for example having the right number of spaces for indentation (e.g. for list items) and such. This part might be a bit fragile. But hopefully, with some agreed standard formatting this could work as well.

Btw, for simple override use cases the syntax would be identical with the definitions right?

test-profile:
  - duration: 24h

Priority

As part of the discussions it was proposed we should have a clear definition of the individual priorities with which the profiles would be applied to make it clear to the user what to expect. One of the suggestions was that more specific (e.g. metadata close to the individual test) should override general values.

This templating approach means that the profile (or last of the applied profiles if we consider applying multiple ones) will always have the last word, right? So, for example, if the test would like to state that avc denial is expected, the profile has to be very carefully written in order to respect this.

Defaults

Related to the priority question: How would we define a default value which could be overriden by tests? Just compare with the known hard-coded tmt default? For example how to set the default test duration to 1h for all tests which do not have a custom duration defined?

test-profile:
  - duration: |
      {% if VALUE == 5m %}
      1h
      {% else %}
      {{ VALUE }}
      {% endif %}

I guess something like this should be included in the list of basic use cases to be supported.

Auditability

Completely agree with @thrix's point, that we need to make it really simple to understand / explore / find out how the profile was applied and which changes it caused. This is crucial, it's already a significant amount of magic.

And we also need to make it super-simple to reproduce the ci environment on user laptop. Perhaps allowing to directly reference remote profiles from the tmt run command so that users do not have to download them? Probably a bit off-topic here...

Overall

All in all I like the approach. Just would like to clarify the expectations around the priorities and default values as mentioned above. Plus agree it would be good to confirm this covers the use cases we already know about. Perhaps we could/should step out of the MVP scope here for a bit and double check the same approach would work for plans as well? Not deep-dive, just to ensure we don't close some important door too early.

May 15 '25 22:05 psss

Thinking about the ORIGINAL example: I guess there might be also some pitfalls, like for example having the right number of spaces for indentation (e.g. for list items) and such. This part might be a bit fragile. But hopefully, with some agreed standard formatting this could work as well.

That should be less risky than it might seem: under the hood, ORIGINAL and other variables, when emitted into the rendered string, are rendered as JSON objects. It will not be visible when logged by tmt, because tmt treats the string as YAML and will output nice formatting, but the indentation will often not be that critical.

Btw, for simple override use cases the syntax would be identical with the definitions right?

test-profile:

duration: 24h

Correct.

Priority

As part of the discussions it was proposed we should have a clear definition of the individual priorities with which the profiles would be applied to make it clear to the user what to expect. One of the suggestions was that more specific (e.g. metadata close to the individual test) should override general values.

This templating approach means that the profile (or last of the applied profiles if we consider applying multiple ones) will always have the last word, right?

Correct, as implemented in the example PR, the profile is applied after Test instances materialize from fmf nodes. Hence, the VALUE representing the current value of the field.

It is, of course, possible to move the point where the profile is applied: the demo does it after normalization of test instances. Another suitable point would be before Test instances are born, but that has its own set of caveats:

it would see whether the key exists or not, i.e., it would immediately be able to tell whether duration was set or not, and have a much easier position to implement "update missing" behavior;
it would face much diverse input, especially for keys that support different input forms:
```
tag: foo
tag:
  - foo
```
This way would put more stress on profile development and maintenance, because it would need to work with values that are not normalized yet.

Another way would be tmt not even allowing profile to apply if the field is set by user. It would still require tracking of the value origin (see below), but, it would not let the profile to touch the field by skipping instructions for fields that were set by user. In other words, duration instructions would be skipped if test had duration key set. Again, that has its own set of issues as well, such a rule would not work well with keys like check that have inner structure, keys that are composed from smaller bits. In the case of check, we do not care about its default, that's an empty list, we care about whether the actual value contains a special object with how == avc.

So, for example, if the test would like to state that avc denial is expected, the profile has to be very carefully written in order to respect this.

I'd argue it needs to be carefully written in any case, but yes, as demonstrated in the AVC check example above, it is the responsibility of the profile to decide and include the original value if necessary.

Defaults

Related to the priority question: How would we define a default value which could be overriden by tests? Just compare with the known hard-coded tmt default?

That has been a pain point of CLI and fmf interaction: duration: 5m in test metadata is explicit yet same as the default, and simple comparison will identify it as the default value, open to overwrites. Without tracking of the origin of the value, this behavior cannot be reasonably implemented. Click offers "value source" we use when incorporating CLI input of --update/--update-missing, and we would need the same beyond that so tmt would be able to tell whether a value has been defined by user or supplied by field's default or default_factory arguments.

The example PR applies profile after normalization, when all Test fields are set, including those that were not provided by fmf or CLI, those are set to their corresponding default values. To safely recognize whether the value is "true" or "just default", the normalization layer will need to learn to track the origin and expose it to the rest of the code.

For example how to set the default test duration to 1h for all tests which do not have a custom duration defined?

test-profile:

duration: | {% if VALUE == 5m %} 1h {% else %} {{ VALUE }} {% endif %}

As explained above, this will fail for test that do have duration: 5m in their metadata if we use a mere comparison for recognizing default values.

I guess something like this should be included in the list of basic use cases to be supported.

Absolutely. It will cost us more in terms of time and implementation, of course.

Overall

All in all I like the approach. Just would like to clarify the expectations around the priorities and default values as mentioned above. Plus agree it would be good to confirm this covers the use cases we already know about. Perhaps we could/should step out of the MVP scope here for a bit and double check the same approach would work for plans as well? Not deep-dive, just to ensure we don't close some important door too early.

Please, throw in some of these examples.

My view: there are different ways where the profile could be applied, doing so as the last step and giving profiles access to the original value seems like the best option to me. It puts more responsibility on profile maintainers, which I think is acceptable: they should be careful and test things by default anyway. It is much less open to exceptions, coming out as the simplest in terms of rules, and gives profiles a lot of flexibility without introducing too much magic. It is, however, struggling with set-by-user-to-default-value keys, but that has been common in tmt codebase; things improved with --update-missing, and teaching the normalization layer to track the origin of key value would be the solution; we can then decide whether to even run the instruction, or whether pass the "is default" flag into the template (latter for me, less exceptions, the former would need different behavior for duration and check).

Edit: the CLI mentioned above might sound like being important in the case of plan metadata only, e.g. when the profile needs to play nicely wth --insert or --update while trying to modify prepare steps. But while we do not have --update for tests, execute --duration is a CLI input mangling test key, and there is a call for --adjust-tests to be available. These introduce "CLI" as a valid source for Test field values, and together with fmf and the default value it could easily turn messy without proper tracking. --update/--update-missing certainly did, that's why there are CLIInvocations, option_sources, and _apply_cli_invocations() carefully tieing it all into one piece.

May 16 '25 09:05 happz

Thinking about the ORIGINAL example: I guess there might be also some pitfalls, like for example having the right number of spaces for indentation (e.g. for list items) and such. This part might be a bit fragile. But hopefully, with some agreed standard formatting this could work as well.

That should be less risky than it might seem: under the hood, ORIGINAL and other variables, when emitted into the rendered string, are rendered as JSON objects. It will not be visible when logged by tmt, because tmt treats the string as YAML and will output nice formatting, but the indentation will often not be that critical.

Now I'm a bit confused: So when writing the template I don't need to be careful about where exactly place the variable? For example just adding an extra check:

test-profile:
  - check: |
      - how: foo
        some: extra

        {{ ORIGINAL }}

Here it would not matter where exactly the {{ ORIGINAL }} is placed?

May 16 '25 09:05 psss

Thinking about the ORIGINAL example: I guess there might be also some pitfalls, like for example having the right number of spaces for indentation (e.g. for list items) and such. This part might be a bit fragile. But hopefully, with some agreed standard formatting this could work as well.

That should be less risky than it might seem: under the hood, ORIGINAL and other variables, when emitted into the rendered string, are rendered as JSON objects. It will not be visible when logged by tmt, because tmt treats the string as YAML and will output nice formatting, but the indentation will often not be that critical.

Now I'm a bit confused: So when writing the template I don't need to be careful about where exactly place the variable? For example just adding an extra check:

test-profile:

check: |

how: foo some: extra

{{ ORIGINAL }} Here it would not matter where exactly the {{ ORIGINAL }} is placed?

Correct, I mistook it for a different example. The indentation of {{ ORIGINAL }} would indeed matter.

May 16 '25 11:05 happz

So, what are the next steps here and now?

check out how tmt can start tracking key value origin to make it exposed to profiles, which would resolve the default/not default decision making process
add --diff-ish option to tmt test export so we can play with the logging
check out whether this "changelog" could be written into an actual file
check out how tmt can expose the tmt profile name in results
check out how tmt can expose the tmt profile name in test/phase environment
add more usecases and examples, not limited to tests

May 16 '25 11:05 happz

That looks like a very good outline! Agreed. Perhaps a doc would be good. I experimented with Gemini to provide an overall summary of the proposed concept and I'd say it did a pretty decent job: I've pasted the output to the outline issue:

#3692

@happz, could you please have a look and adjust any possible mistakes or omissions? Seems we three have a consensus. Perhaps, we could announce the draft, just to make sure there are no serious objections or concerns?

May 16 '25 12:05 psss

That looks like a very good outline! Agreed. Perhaps a doc would be good. I experimented with Gemini to provide an overall summary of the proposed concept and I'd say it did a pretty decent job: I've pasted the output to the outline issue:

Outline the default workflow configuration implementation #3692

@happz, could you please have a look and adjust any possible mistakes or omissions? Seems we three have a consensus. Perhaps, we could announce the draft, just to make sure there are no serious objections or concerns?

LGTM

May 19 '25 08:05 happz

allow TMT_POLICY_NAME through TF API: https://gitlab.com/testing-farm/infrastructure/-/merge_requests/983 (review & deployment pending)
bundle policies into TF worker images: https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/952 (review & release pending)
integration test for TF is blocked by the first PR
general usage is blocked by items above, TF release and and verification everything works after that. Release is supposed to happen next week, and I plan to be on PTO the week after.

Jun 27 '25 11:06 happz

TMT_POLICY_NAME is now accepted by TF
TMT_POLICY_ROOT is set and exposed to TF pipelines and tmt
policies are not yet bundled into the image, pending TF release: https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/952
I tested the functionality with the custom image from the MR above, worked like a charm

Next steps:

get the policies into TF worker images
once deployed, test it with some real-life jobs
find issues, report, fix, rinse and repeat
consider a better way of policy delivery method, updating worker images after every commit to policies repository seems cumbersome (good enough for now...)

Jul 03 '25 20:07 happz

Support for the default workflow configuration

Templating-Based Solution – Initial Observations

General Impressions

Design Considerations for Templating

Key Decisions Before First Release

Final Thoughts

Key Decisions Before First Release

Final Thoughts

Key Decisions Before First Release

Final Thoughts

Simplicity

Priority

Defaults

Auditability

Overall

Priority

Defaults

Overall