add script to compare git to rpm repo

[evgeni]

[evgeni]

% python compare_rpm_repo.py                                     
foreman
In repo but not in git:
[]
In git but not in repo:
[]
katello
In repo but not in git:
[]
In git but not in repo:
[('rubygem-qpid_proton', '0.37.0'), ('rhel8-kickstart-setup', '0.0.2'), ('jsoncpp', '1.8.4')]
plugins
In repo but not in git:
[('yggdrasil', '0.2.0')]
In git but not in repo:
[('yggdrasil-worker-forwarder', '0.0.1')]
client
In repo but not in git:
[('qpid-proton', '0.32.0')]
In git but not in repo:
[]

% python compare_rpm_repo.py --staging
foreman
In repo but not in git:
[]
In git but not in repo:
[]
katello
In repo but not in git:
[]
In git but not in repo:
[('rubygem-qpid_proton', '0.37.0'), ('jsoncpp', '1.8.4'), ('rhel8-kickstart-setup', '0.0.2')]
plugins
In repo but not in git:
[('yggdrasil', '0.2.0')]
In git but not in repo:
[('yggdrasil-worker-forwarder', '0.0.1')]
client
In repo but not in git:
[('qpid-proton', '0.32.0')]
In git but not in repo:
[]

• rhel8-kickstart-setup fix is in https://github.com/theforeman/foreman-packaging/pull/9231
• yggdrasil-worker-forwarder fixed in https://github.com/theforeman/foreman-packaging/pull/9236
• yggdrasil is special (it's packaged for client but used in plugins and client)
• qpid-proton is special (it's packaged for katello but used for katello and client)
• jsoncpp is build-time only dep (see https://github.com/theforeman/foreman-packaging/pull/6229)
• rubygem-qpid_proton fix is in https://github.com/theforeman/foreman-packaging/pull/9255

[ehelms]

A couple of questions:

1. Does this catch things that obal verify-koji-tag does not?
2. This creates a(nother) split in where you go and what you use to do one thing or another.
3. The name of the script is not as obvious to me what repositories are being compared to whom. If I ran this on the rpm/3.6 branch I would expect it to compare 3.6 repositories?

[evgeni]

obal verify-koji-tag

damn I forgot about that one!

The name of the script is not as obvious to me what repositories are being compared to whom. If I ran this on the rpm/3.6 branch I would expect it to compare 3.6 repositories?

and it will!

[evgeni]

so obal verify-koji-tag compares the list of packages in git with the one that is allowed (koji add-pkg) in the tag (using koji list-pkgs --tag ...)

the script here compares the list of packages and their versions between git and whatever is on our repos, the reasoning is:

• just because a package is allowed (pkg-add) doesn't mean there is a build
• just because a package is built doesn't mean it shows up in the repo

The first one could be solved by using koji list-tagged --latest instead of koji list-pkgs. The second one can't be fixed by koji itself, but it might be not that much of a problem overall.

I don't mind too much where the script lives (even tho I like that it doesn't need a koji cert and can be run via a github action daily), but I want to be able to compare packages with versions :)

[evgeni]

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

[ehelms]

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

Ahh this happened when we switched away from tito, and now qpid-proton is defined for katello packages (https://github.com/theforeman/foreman-packaging/blob/rpm/develop/package_manifest.yaml#L507) but not getting the client tags associated to it. We could move it to it's own special little group and then make that a child of katello and client?

[evgeni]

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

Ahh this happened when we switched away from tito, and now qpid-proton is defined for katello packages (https://github.com/theforeman/foreman-packaging/blob/rpm/develop/package_manifest.yaml#L507) but not getting the client tags associated to it. We could move it to it's own special little group and then make that a child of katello and client?

Not sure how that would work as obal can add missing tags, but when you build the initial tag is done by koji, so effectively you'd have to obal build twice to get both tags? But if that'd work, I'm down.

Same applies for yggdrasil that goes to plugins and client btw :)

[ehelms]

Not sure how that would work as obal can add missing tags, but when you build the initial tag is done by koji, so effectively you'd have to obal build twice to get both tags? But if that'd work, I'm down.

Hrmm, you are right, it's not built to handle same build different tags in the list of tags. However, I think because we run the tags in a loop, Ansible will run each chunk serially? So if it finds the build it will tag it?

[evgeni]

Mhh, you're right. Reading the code, this is what should happen. I agree. I'll try to add an explicit test for that behaviour

Edit: https://github.com/theforeman/obal/pull/341

[evgeni]

TIL: there is also obal check :exploding_head:

[ehelms]

TIL: there is also obal check exploding_head

I mean, I tried to close the gaps and add all the things to it :smile: