tmt should switch from Beaker task repo to gitlab as a source for libraries not found on github

[martinky82]

Installing the rpms from Beakr tasks repo fails in Image Mode:

          Running scriptlet: restraint-rhts-0.4.14-1.el10.x86_64                214/226 
          Installing       : distribution-distribution-Library-LibrariesWrapp   215/226 
        error: failed to open dir mnt of /mnt/: cpio: mkdir failed - File exists

          Installing       : distribution-distribution-Library-Log-1.0-34.noa   216/226 
        error: unpacking of archive failed on file /mnt/tests: cpio: mkdir failed - No such file or directory
        error: distribution-distribution-Library-LibrariesWrapper-1.0-12.noarch: install failed
        error: failed to open dir mnt of /mnt/tests/distribution/Library/: cpio: mkdir failed - File exists

Combined with: https://github.com/teemtee/tmt/issues/4196 it completely breaks IM testing for perl and most likely many more components.

My proposal is: if a library is not found on github, search for it on gitlab.com/redhat/rhel/tests. This looks not to be very hard to implemene and it will effectively replace the dnf/beaker tasks fallback. We will need to move away from Beaker tasks anyways.

[thrix]

Just noting that discovering from that repository will require authentication, what is not a big deal, bu the user will need to be aware when running tests locally. In Testing Farm it wlll just work

[martinky82]

I guess, the same gitlab project being the default test storage means the user ill be authenticated in most cases anyways.

[LecrisUT]

Seems related to #477. Should we close this as a duplicate to that, and see how we can support it with the context there.

[martinky82]

I guess the #477 is about ability to define the fallback location when I run tests. This one is about the default fallback location being Gitlab (the second fallback to beaker tasks may still remain but I don't think there is a library in Beaker tasks that doesn't have its source on Gitlab.

[LecrisUT]

Wouldn't a simple regex find-and-replace be sufficient to move them after we have a specific language for how to define git forge independent library()? Something like library\((?P<beakerlib_lib>.+)\) -> library(gitlab.com:redhat/rhel/tests/$beakerlib_lib)?

[LecrisUT]

Also to note that we do support additional locations via

require:
  - type: library
    url: https://github.com/beakerlib/openssl
    name: /certgen

You could make a similar regex find-and-replace to move them all to gitlab.com:redhat/rhel/tests/. Would also that work or would it explicitly need this hard-coded fallback?

[kkaarreell]

Are there other RH-specific locations and other specifics already hard-coded to tmt? I am trying to figure out how adding to tmt something "RH internal" which won't work for anyone else makes sense.

[martinky82]

Well, there are two requests here:

1. provide a system level fallback (no, updating every single test is not a question, there are hundreds of them)
2. don't hardcode RH-specific bits in tmt

Those two are both valid and also kind of conflicting.

I wonder, would it be possible to add a mechanism to use more than one git repo as a source of libraries?

• by default, it would be github/beakerlib
• gitlab/.../tests passed as an 2nd level location via variable
  • in this case we need to take into account different directory layout (same as with rpm though so I assume, so rlImport will work)

Example: test reqs library(comp/lib)

1. git clone gitlab/beakerlib/comp
2. if not possible and $alternative_git is set, git clone $alternative_git/.../tests/comp
3. rpm falllback (as long as beaker taskrpms are around) - with $alternate_git being gitlab/.../tests, it should never be used as all libraries in Beaker task rpms have their sources there or more recent versions on github

Would that be acceptable?

[LecrisUT]

Do you need these to be run in testing-farm or locally?

[martinky82]

TF

[martinky82]

My direction of thinking is: we have test profiles. So we have a way to assign $alternative_git the RH-specific value for all of RHEL testing w/o touching each and every test - just add it to the rhel-ci test profile. But, tmt needs to know what to do with it.

And this fix would be generic.

[kkaarreell]

no, updating every single test is not a question, there are hundreds of them

Claude, create a bash script that recursively finds all main.fmf file under a given repository and modifies it like this.

For YAML list items like
 - library(repository_name/library_name)
it checks if the "repository_name" is available in project https://github.com/beakerlib/ and if yes, it will preserve the record.
If the repository is not there, it will be transformed into a record like this:
 - type: library
   url: https://github.com/beakerlib/repository_name
    name: /library_name

... 4 minutes of conversations of doodlibuudleding...

I've created the bash script modify_main_fmf.sh that will:

  Features:

  1. Finds all main.fmf files recursively in the provided repository path
  2. Checks GitHub availability using the GitHub API to verify if beakerlib/<repository_name> exists
  3. Preserves existing format if the repository exists on GitHub
  4. Transforms to new format if the repository doesn't exist with:
    - type: library
    - url: https://github.com/beakerlib/repository_name
    - name: /library_name
  5. Maintains indentation from the original file
  6. Provides colored output showing what was found and modified

  Usage:

  ./modify_main_fmf.sh <path_to_repository>

So let's try:

$ cat main.fmf 
require: 
 - aide
 - rsync
 - library(openssl/certgen)
 - library(something/unavailable)
$ cat a/main.fmf 
require: 
 - rsync
 - library(something/unavailable)
 - library(scl-utils/scl)
 - aide
$ ./modify_main_fmf.sh .
Scanning for main.fmf files in: .
==========================================

Processing: ./a/main.fmf
  Found library entry: scl-utils/scl
  ✓ Repository beakerlib/scl-utils exists - preserving
✓ File processed (no changes needed)

Processing: ./main.fmf
  Found library entry: openssl/certgen
  ✓ Repository beakerlib/openssl exists - preserving
  Found library entry: something/unavailable
  ✗ Repository beakerlib/something not found - transforming
✓ File modified successfully

==========================================
Processing complete. Total files processed: 2
$ cat main.fmf 
require: 
 - aide
 - rsync
 - library(openssl/certgen)
 - type: library
   url: https://github.com/beakerlib/something
   name: /unavailable
$ cat a/main.fmf 
require: 
 - rsync
 - type: library
   url: https://github.com/beakerlib/something
   name: /unavailable
 - library(scl-utils/scl)
 - aide

I can mail you the script privately if you are interested... Please, just fix those tests.

[martinky82]

The tests ain't broken. You can email the script to me, would you email it to everyone affected and see that they run it instead of:

adjust:

• enabled: false
• when: deplyoment-mode = image
• continue: false

What I am kindly asking for it configurable second git repo as as source of libraries. Nothin RH-specific, somethung that should've been implemented from the begining, frankly. If it's such a fuss to implement, I WILL PROVIDE A PATCH. I can run Claude to generate one, anyways. Better 'fixing' such an issue in one place than in several... dozens or hundreds even? And it;s not even a fix, it's a feature.

[kkaarreell]

Tests are broken. They rely on a fallback behavior and should have been updated like 3 years ago to not to depend on a beaker task library.

[martinky82]

The tests use the library(x/y) format of requirement, which was never deprecated and is documented as the DEFAULT in tmt documentation and works with githob. The tests do not take care of how the requirement (which is correct) is fulfilled. I am really NOT in favour of boilerplating the same stuff over and over again.

[LecrisUT]

Do you need these to be run in testing-farm or locally?

TF

The issue is that even if it is fixed there, you would have to inject the environment variable or equivalent in all of the executions of testing-farm, and if anything moves, those would have to move with it as well. These would not be long-term solutions, only clutches to help you migrate, but the migration is to use the

 - type: library
   url: https://github.com/beakerlib/something
   name: /unavailable

format. That is the only format that is portable and that would allow other peoples to run these tests locally.

Migrating the format for those broken tests is indeed a chore, but that is also part of the steps to fix the tests.

[martinky82]

https://github.com/teemtee/tmt/pull/4355

[kkaarreell]

Better 'fixing' such an issue in one place than in several... dozens or hundreds even? And it;s not even a fix, it's a feature.

It would be very unfortunate feature. It makes successful tmt test plan execution dependent on some weird variable whose magic value won't be stored in fmf metadata and which everyone willing to run the test in a container or own virtual machine would have to know and configure. And despite you claim it is a generic and non RH specific feature, it is clear that it opens door for library name conflicts where different tests could depend on distinct libraries of the same name and only the value of this variable would decide which one will be used. It is a time bomb that will hit users again in the future. We now have tools to migrate test metadata to the universal format for tens or hundreds of repos in less than 15 minutes, to me this is truly a no-brainer.

[thrix]

I am also tending just to fix all tests relying on fallback behavior to beaker tasks. I believe all tests using that should be now easily discoverable via:

https://sourcebot.dno.corp.redhat.com/~/search?query=content%3Alibrary%5C(%20file%3Afmf

Let's send out an email clearly informing users the new tests using libraries should use the ones from github, or use a direct git reference if it is not present. It is a known fact beaker tasks are going away sooner or later. Provide them a vibe coded script Karel has around to help with the migration.

Then we can spend our precious development time on something else. I do not mind personally adding one or more location to the fallback, but there clearly there are more. And I am aligned with the idea that it is worth complicating the code with providing this list of fallback git repositories or something like that.

[martinky82]

Let's put it another way: what I want here to implement is additional configurable git repo(s). It can be done via new variable (see the patch) or treating the DEFAULT_REPOSITORY_TEMPLATE as a list and make it overridable from env. It surely wouldn't hurt and it's not up to tmt to care about if and what users stuff in that env variable. Since 99% of the 'fallback' libraries are in one location (gitlab/.../tests/), it can be considered RH-internal standard, not a fallback and we have means to enable the standard globally (test profiles, anyone?). However, with that in place, the rpm fallback can be removed. I still think it would be kind of a boilerplate to copy the same git path over and over. However, if you want to declare the 'library(x/y)' deprecated, feel free to do so officially and notify all the users to change their fmf to accomodate it. Mind you, the problem here is not new tests, but the tests already existing. It's nice to have vibe-coded tool to do that but review will still be required (trust but verify). I guess tasking people with such a chore will fill their hearts with joy and excitement. Or, maybe... just fix the 'we don't do prepare on guest in IM' issue.

[LecrisUT]

library(x/y) is not deprecated, and we explicitly document its usage as

When referencing beakerlib libraries it is possible to use both the backward-compatible syntax library(repo/lib) which fetches libraries from the default location as well as provide a dictionary with a full fmf identifier. For the latter case specify type: library.

(i.e. only from beakerlib on github)

Please discuss one topic that was raised multiple times:

It makes successful tmt test plan execution dependent on some weird variable whose magic value won't be stored in fmf metadata and which everyone willing to run the test in a container or own virtual machine would have to know and configure

This is a very important topic to consider how viable it is to provide such functionality vs requesting users to patch their test files.

[kkaarreell]

However, if you want to declare the 'library(x/y)' deprecated, feel free to do so officially and notify all the users to change their fmf to accomodate it. Mind you, the problem here is not new tests, but the tests already existing. It's nice to have vibe-coded tool to do that but review will still be required (trust but verify). I guess tasking people with such a chore will fill their hearts with joy and excitement. Or, maybe... just fix the 'we don't do prepare on guest in IM' issue.

It is not deprecated syntax. It is RPM symbol name that originates in Beaker task library RPMs and tmt have implemented its special handling for one location only in order to support users migrating their beakerlib libraries to that one particular repository location. The fact that it is just one location make the tmt behavior deterministic and it means it will work the same way for any user running those tests whatever way and whatever pipeline.

Users of libraries that are not hosted on that one supported GitHub location had 3 years (!!!) to modify test metadata so that they won't depend on RPMs from the Beaker tasks repository and the removal of Beaker tasks repository is not even in tmt project responsibility or decision. So let's be clear that this is not about deprecating tmt functionality (case 1) but about the later.

[martinky82]

Users of libraries that are not hosted on that one supported GitHub location had 3 years (!!!) to modify test metadata so that they won't depend on RPMs from the Beaker tasks repository and the removal of Beaker tasks repository is not even in tmt project responsibility or decision. So let's be clear that this is not about deprecating tmt functionality (case 1) but about the later.

Did anyone tell them yet (or 3 years (!!!) ago)? I failed to notice.

[martinky82]

This is a very important topic to consider how viable it is to provide such functionality vs requesting users to patch their test files.

I agree. It is important to consider.

[martinky82]

Anyways, most problematic now seems to be the epel library which, even if it lives in GitHib, requires epel-internal library which does not and will not (obviously). I guess, it's not very good practice for a GitHub library to require library that is not on GitHub. Perheaps, with removing of that req most of the problem will just go away and there will be just few bits here and there to deal with.

FTR: https://github.com/beakerlib/epel/issues/9

[martinky82]

You say: FIX YOUR TESTS. But, what if the tests are fine: they actually DO require only libraries that are in githib... But the libraries have requirements on libraries that are not (epel/epel -> distribution/epel-internal)? Is it OK for a publicly accessible lbrary to expose git locations that are considered internal?

FTR: that was my case with IM: vsftpd/basic library pulled by chain of lib/lib dependencies 7 other libraries which were not in github and made my testing to fail. It was used in single test, I disabled it from IM. Apart from that, I am not affected anymore.

[LecrisUT]

You say: FIX YOUR TESTS. But, what if the tests are fine: they actually DO require only libraries that are in githib... But the libraries have requirements on libraries that are not (epel/epel -> distribution/epel-internal)? Is it OK for a publicly accessible lbrary to expose git locations that are considered internal?

That does sound like the real source of the issues. Indeed that is where it should be fixed. The key point here is that you successfully uncovered this through trying to fix your test and finding this intermediate dependency. This would have been quite hard to traceback with any of the other approaches proposed, not even the logging proposal would have helped here iiuc. Again thank you for uncovering this issue and reporting to the relevant repo.

[thrix]

FTR testing farm now can install distribution/epel well as we dropped beaker-tasks so the priority of this issue should be a lot lower.