[Feature Request] Add filtered Module Stream metadata to CLs

[lumarel]

As discussed in the today's Community Hours I would like to propose a improvement to the Module Stream implementation (for Enterprise Linux 8, 9, Fedora 29, 30, 31, 32, 33, 34, 35, 36).

Right now the situation is, you can sync the repos to Uyuni, but if you want to use the Module Streams you can only add them in synced (Base) repos. To make Uyuni deal with the update process, it is needed to create Content Lifecycle Environments. The issue there is, that as soon as a machine is in the CL environment, it doesn't see any Module Stream data anymore, as these just get stripped out there, this creates at least three major problems:

• You are not able to add any Streams to the machine, it will be able to access the packages, but it won't be able to add the Stream (there are at least a handful of installers which will expect to be able to check/enable the Streams)
• There are several Streams which don't have the default enabled flag set, which means the packages of these are normally not accessible before manually enabling the Stream (so creates unknown behavior)
• There is also package group information in the Streams metadata

I know that making Uyuni Module Streams agnostic looks like a lot of work, and I also think that it would be, but as discussed, there might be a rather resource saving way to implement something so dnf doesn't get completely confused.

I'm not really a programmer, but for me it sounds like the CL creator (only for the initial build process) would need a function to be able to take the ingest modules.yaml file and Filter data which gets added to the CL, and just filters the Streams in this file which are present as a Filter.

So to visualize it a bit the original modules.yaml has 2 document types in it, modulemd-defaults and modulemd, these always look like this:

---
document: modulemd-defaults
version: 1
data:
  module: container-tools
  stream: "rhel8"
  profiles:
    1.0: [common]
    2.0: [common]
    3.0: [common]
    4.0: [common]
    rhel8: [common]
...
---
document: modulemd
version: 2
data:
  name: container-tools
  stream: "1.0"
  version: 8050020211109140251
  context: faa19cc5
  arch: x86_64
  summary: Stable versions of podman 1.0, buildah 1.5, skopeo 0.1, runc, conmon, CRIU, ...
  description: >-
    Stable versions of podman 1.0 , buildah 1.5, skopeo 0.1, runc, conmon, CRIU, Udica, ...
  license:
    module:
    - MIT
    content:
    - ASL 2.0
    - GPLv2
    - GPLv3+
  xmd: {}
  dependencies:
  - buildrequires:
      go-toolset: [rhel8]
      golang-ecosystem: [1.0]
      platform: [el8.5.0]
    requires:
      platform: [el8]
  references:
    community: https://github.com/projectatomic
    documentation: https://projectatomic.io
    tracker: https://github.com/projectatomic
  profiles:
    common:
      rpms:
      - buildah
      - container-selinux
      - ***
  api:
    rpms:
    - buildah
    - container-selinux
    - ***
  buildopts:
    rpms:
      macros: >
        %_with_ignore_tests 1
  components:
    rpms:
      buildah:
        rationale: Primary component of this module
        ref: 7d7e010bae87dd13000b3c9c633527beba1f1066
        arches: [aarch64, i686, x86_64]
    ***
      ***
        ***
        ***
  artifacts:
    rpms:
    - buildah-0:1.5-8.gite94b4f9.module+el8.5.0+681+c9a1951f.src
    - buildah-0:1.5-8.gite94b4f9.module+el8.5.0+681+c9a1951f.x86_64
    - ***
...

There is only at most 1 modulemd-defaults item and at least 1 modulemd item for every Module Stream. (if there is no default Stream set, there is no modulemd-defaults available) All of them start with --- and end with ...

So filtering for the set modules would mean removing the not present profiles from the dictionary in the modulemd-defaults item (if present), and completely removing the modulemd items, which are filtered out.

And the file which will get emmited from this would need to be put in the CL repo structure and referenced in the repomd.xml file as data type modules (I guess the metadata for the other stuff in the repomd.xml file is already autogenerated anyways as this changes as well)

I hope this is understandable enough and makes sense to implement it in this way, if not I'm also fine with a better solution! 🙂

If any further knowledge is needed from my side I would really like to help! Thank you very much!

[admd]

@cbbayburt fyi

[cbbayburt]

@lumarel thanks for the well-written description. This sounds totally useful and doable. I believe this could also save us a lot of trouble with some unexpected issues in the future.

[lumarel]

One afterthought I had as well now, in the case someone selects something different than the value in data.module in the modulemd-defaults the whole item should be removed as that's not really useful to have a reference in the nowhere ^^

[lumarel]

As Rocky Linux 9 (actually all EL9 distros) got the first modules now, this gets more interesting again...

[admd]

@lumarel any major differences from EL8 when it comes to modules?

[lumarel]

@admd Nope, still the same old module stream standard :) https://docs.fedoraproject.org/en-US/modularity/

[lumarel]

The only thing is, EL9 only has 4 non default module streams, which can be activated to get a later version of the 4 applications Btw, EL8 also has a few module streams without a default stream (there is also a problem currently, as you can't filter out streams if no default is available)

[cbbayburt]

there is also a problem currently, as you can't filter out streams if no default is available

@lumarel what do you mean by this? In the current implementation, you should be able to filter any stream even if there's no default defined.

[lumarel]

what do you mean by this? In the current implementation, you should be able to filter any stream even if there's no default defined.

@cbbayburt Since RHEL9.1 (and further down Rocky Linux 9.1) there are now 4 module streams:

• maven:3.8
• nodejs:18
• php:8.1
• ruby:3.1

https://gitlab.com/redhat/centos-stream/release-engineering/module-defaults/-/tree/c9s (this is from CentOS Stream) https://git.rockylinux.org/rocky/rocky-module-defaults/-/tree/r9.1 (ignore the java streams, they are for the platform bootstrapping) notice the missing data.stream variable

None of these have a default set, which means the packages will be ignored as long as no one runs a dnf module enable <stream-name>, this means the older version which is directly accessible will be used by default. But Uyuni is unable to filter out the packages if there is no CL filter defined, and I can't define a filter, because there is no default.

The same issue occurs for EL8, where there are some module streams, which don't have a defined default (eclipse is the only one there), as far as I remember the modulemd-defaults directives are missing at all in the metadata there.

[lumarel]

And here is the reason why I can't add a completely excluding filter:

or also with just an incorrect value:

[cbbayburt]

I think I still didn't quite get what you mean.

If a module doesn't have a default stream defined, the only effects in CLM are:

• In "Create filter" form, the "Stream" field won't be prefilled with the default stream. You must enter the stream manually
• You won't be able to use "AppStreams with defaults" filter template

Default streams are only used in the UI to prefill these values. In the actual operation, default values are irrelevant. So in the end, you should be able to add a filter for the module by entering the module and stream fields manually.

[lumarel]

@cbbayburt The 2 errors from the last message come from the add filter to CL operation, tbh this is somewhat irrelevant, because I think there has either exist documentation how to completely remove a module stream (even if a default exists), and or a option to remove a module stream completele, or it has to happen a breaking change, which by default removes all module streams without a set filter.

Furthermore, this is actually a non-target of this feature request. (should have moved that discussion to another issue a few messages ago)