uyuni Automate changelog handling for container images

What does this PR change?

The idea is to integrate dockerfiles and helm charts into our rel-eng workflow. This consists of two parts:

This PR is coupled together with this patch for tito. the tito was extended so that it can now deal with dockerfile and Chart.yaml when calling tito tag command. So it makes possible the tagging, therefore the reporting with tito report or tito-wrapper -c works automatically once the first tag gets checked into the git.

tito now tries to read a dockerfile or a Chart.yaml file if there's no .spec file in place.
once a dockerfile or a Chart.yaml is detected, tito performs its usual business, i.e. bumps the version, saves the new version into the mentioned file, updates the changelog and checks in everything into the git.

There're changes to this repository so that tito can find its way to a corresponding directory. We have:

entries under rel-eng/packages where file name is the name of the package and the content is pointing to a directory in the repo.
there're tito.prop files pointing to the new tagger from the patch above.
for Chart.yaml there's version: 4.3.1-1 entry and for Dockerfile the LABEL org.opencontainers.image.version=4.3.1-1 one. Tito reads version from and updates them in there.

So hopefully the usual workflow should apply also to this part of the codebase.

GUI diff

No difference.

[X] DONE

Documentation

No documentation needed: only internal and user invisible changes
[X] DONE

Test coverage

No tests: rel-eng things
[X] DONE

Links

Fixes https://github.com/SUSE/spacewalk/issues/17786

[X] DONE

Changelogs

Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository

If you don't need a changelog check, please mark this checkbox:

[ ] No changelog needed

If you uncheck the checkbox after the PR is created, you will need to re-run changelog_test (see below)

Re-run a test

If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run:

[ ] Re-run test "changelog_test"
[ ] Re-run test "backend_unittests_pgsql"
[ ] Re-run test "java_pgsql_tests"
[ ] Re-run test "schema_migration_test_pgsql"
[ ] Re-run test "susemanager_unittests"
[ ] Re-run test "javascript_lint"
[ ] Re-run test "spacecmd_unittests"

Aug 24 '22 16:08 avshiliaev

@juliogonzalez So I think it basically solves the problem the way you described it here, unless I've missed something :smile:

Aug 24 '22 16:08 avshiliaev

@cbosdo I've removed the lines from (1) here, since this is being added by tito (2) here.

Dec 09 '22 16:12 avshiliaev

@cbosdo @juliogonzalez Hey guys, I've remove some unused code from the shell script as discussed, also I've removed some weird changes to the changelogs files I've made by mistake while testing the thing. I've also tested everything once again with tito tag

Jan 13 '23 15:01 avshiliaev

Tagging works now, but I am still getting issues if I need to retag..

Steps to reproduce:

tag proxy-helm. It will work normally (no need to push stuff)
Add a change to Charts.yml and commit
Try to tag again.
Call tito report --untagged-commits and you will see some packages with a strange error:

 tito report --untagged-commits
Creating output directory: /tmp/tito
Scanning for packages that may need to be tagged...

ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-squid-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-squid-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-squid-image/ no longer exists
------------------------
spacecmd-4.4.5-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
-------------------------------
spacewalk-proxy-4.4.3-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
-------------------------------
spacewalk-utils-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-salt-broker-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-salt-broker-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-salt-broker-image/ no longer exists
----------------------------------
susemanager-schema-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
--------------------------------------
spacewalk-client-tools-4.4.3-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
0facf9aecae464b8a6d86d6b1632df77c9f54498 Translated using Weblate (Macedonian)
e5565ef1014298fb4ad8bb5606341f1ed3579788 Translated using Weblate (Macedonian)
f0bb162ea2a7b3cc52c0c3de61a953a00f999a45 Translated using Weblate (Macedonian)
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-httpd-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-httpd-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-httpd-image/ no longer exists
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-ssh-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-ssh-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-ssh-image/ no longer exists
------------------------------------
uyuni-setup-reportdb-4.4.3-1..HEAD:
0c293b00e44d8f84e21e451940c225d1dac43014 fix (#6531)
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-helm-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-helm-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-helm/ no longer exists
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-tftpd-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-tftpd-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-tftpd-image/ no longer exists

Not sure why that no longer existshappens for some packages?

Feb 13 '23 09:02 juliogonzalez

And as you can see, something is wrong with the tag detection, because:

uyuni-proxy-helm-4.4.2-1..HEAD:
014be5d9276ab232676bcde98d9cf75ab5effeaf test
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-tftpd-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-tftpd-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-tftpd-image/ no longer exists

Which is true, 4.4.1-1 does not exist, but 4.4.2-1` is already tagged:

$ git log
commit 014be5d9276ab232676bcde98d9cf75ab5effeaf (HEAD -> test-container-tagging)
Author: Julio Gonzalez Gil
Date:   Mon Feb 13 10:18:21 2023 +0100

    test

commit 58fe15d87a934fce401076d22ed2432478b85868 (tag: uyuni-proxy-helm-4.4.2-1)
Author: Julio Gonzalez Gil
Date:   Mon Feb 13 10:17:19 2023 +0100

    Automatic commit of package [uyuni-proxy-helm] release [4.4.2-1].

$ cat ../../rel-eng/packages/uyuni-proxy-helm 
4.4.2-1 containers/proxy-helm

$ grep -r 4.4.2 *
Chart.yaml:#!BuildTag: uyuni/proxy:4.4.2-1
Chart.yaml:#!BuildTag: uyuni/proxy:4.4.2-1-build%RELEASE%
Chart.yaml:version: 4.4.2-1
uyuni-proxy-helm.changes:- version 4.4.2-1

Feb 13 '23 09:02 juliogonzalez

Retagging itself seems to be working fine:

$ tito tag
Creating output directory: /tmp/tito
Loaded package specific tito.props overrides
Tagging new version of uyuni-proxy-helm: 4.4.2-1 -> 4.4.3-1
Created tag: uyuni-proxy-helm-4.4.3-1
   View: git show HEAD
   Undo: tito tag -u
   Push: git push origin HEAD && git push origin uyuni-proxy-helm-4.4.3-1
or Push: git push origin HEAD && git push origin --tags

juliogonzalez@localhost:~/uyuni/containers/proxy-helm (test-container-tagging>)$ cat ../../rel-eng/packages/uyuni-proxy-helm 
4.4.3-1 containers/proxy-helm/

juliogonzalez@localhost:~/uyuni/containers/proxy-helm (test-container-tagging>)$ grep -r 4.4.3 *
Chart.yaml:#!BuildTag: uyuni/proxy:4.4.3-1
Chart.yaml:#!BuildTag: uyuni/proxy:4.4.3-1-build%RELEASE%
Chart.yaml:version: 4.4.3-1
uyuni-proxy-helm.changes:- version 4.4.3-1

juliogonzalez@localhost:~/uyuni/containers/proxy-helm (test-container-tagging>)$ git log|head
commit f305f6c241a563709d8e71647cb07c5550ef33f7
Author: Julio Gonzalez Gil
Date:   Mon Feb 13 10:39:43 2023 +0100

    Automatic commit of package [uyuni-proxy-helm] release [4.4.3-1].

Feb 13 '23 09:02 juliogonzalez

@juliogonzalez unfortunately I can't really reproduce the problem. Here's what I'm doing, please correct me if I'm wrong here.

removed tito and reinstalled it from here again, just in case.
I run tito report --untagged-commits and I get among other things the error you mentioned.

ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-helm-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-helm-4.4.1-1..HEAD'

This seems kind of logical, cause there's no tag yet in the repo for helm.

git tag | grep helm

I tag the proxy-helm with tito tag and prove the tag is actually there.

git tag | grep helm

uyuni-proxy-helm-4.4.2-1

I create another report of untagged commits tito report --untagged-commits. There’s no mention of helm anymore with the error 128, cause the tag is there.

git tag | grep helm

uyuni-proxy-helm-4.4.2-1

However, there are still the error messages for things like uyuni-proxy-tftpd-image and others. Still no tags for them.

For example in your case you've posted I see an error related to tftpd, not helm

ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-tftpd-image-4.4.1-1..HEAD -- .

Is this the expected behavior or not? When there's no tag for a package I mean. I assume as soon as we tag them for the first time, they'll get integrated into the normal workflow and the error goes away.

Feb 13 '23 13:02 avshiliaev

Is this the expected behavior or not? When there's no tag for a package I mean. I assume as soon as we tag them for the first time, they'll get integrated into the normal workflow and the error goes away.

I think I see where the problem is. The log is very unclear about it.

-------------------------------
spacewalk-proxy-4.4.3-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
-------------------------------
spacewalk-utils-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-salt-broker-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-salt-broker-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-salt-broker-image/ no longer exists
----------------------------------
susemanager-schema-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements

Problem is this is reporting 4 packages. but it looks like the error about proxy-salt-broker-image belongs to spacewalk-utils

Having:

-------------------------------
spacewalk-proxy-4.4.3-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
-------------------------------
spacewalk-utils-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements
-------------------------------
ERROR: Error running command: git log --pretty=oneline --relative uyuni-proxy-salt-broker-image-4.4.1-1..HEAD -- .

Status code: 128

Command output: fatal: bad revision 'uyuni-proxy-salt-broker-image-4.4.1-1..HEAD'

/home/juliogonzalez/uyuni/containers/proxy-salt-broker-image/ no longer exists
----------------------------------
susemanager-schema-4.4.4-1..HEAD:
1f3f496df10c7f3c326af7f92aff362fc7b5b260 Added make build requirements

Would be clear. But I guess having this problem was always the case, and it's not something caused by your PR.

So I will run more tests tomorrow again. And if everything is OK, I will merge this and the tito SR.

Feb 13 '23 15:02 juliogonzalez

I retested tagging today, and so far works fine.

But building/submitting to OBS seems to be broken. Nothing happens if I specify the packages:

$ export OSCAPI=https://api.opensuse.org/
$ export TEST=1
$ export OBS_PROJ=home:juliogonzalezgil:branches:systemsmanagement:Uyuni:Master2

$ rel-eng/build-packages-for-obs.sh uyuni-proxy-helm uyuni-proxy-httpd-image uyuni-proxy-salt-broker-image uyuni-proxy-squid-image uyuni-proxy-ssh-image uyuni-proxy-tftpd-image
uyuni-proxy-helm uyuni-proxy-httpd-image uyuni-proxy-salt-broker-image uyuni-proxy-squid-image uyuni-proxy-ssh-image uyuni-proxy-tftpd-image
Going to build new obs packages in /tmp/push-packages-to-obs/SRPMS...
======================================================================
Built obs packages:  0
======================================================================

$ rel-eng/push-packages-to-obs.sh uyuni-proxy-helm uyuni-proxy-httpd-image uyuni-proxy-salt-broker-image uyuni-proxy-squid-image uyuni-proxy-ssh-image uyuni-proxy-tftpd-image
uyuni-proxy-helm uyuni-proxy-httpd-image uyuni-proxy-salt-broker-image uyuni-proxy-squid-image uyuni-proxy-ssh-image uyuni-proxy-tftpd-image
Going to update home:juliogonzalezgil:branches:systemsmanagement:Uyuni:Master2 from SRPMS...
No package dir 'SRPMS/uyuni-proxy-helm'
======================================================================
Unchanged packages: 0
Updated packages:   0
======================================================================

Feb 14 '23 15:02 juliogonzalez

Here's a new overview of the changes made so far.

Since we tag containers with tito, it also makes sense to "build" also from this script. They are now getting discovered here anyway
Changed the versioning:

removed the release part from container definitions and from package files.
It's now reflected in the SR, see here for mode detail and here.
There's also a minor change to the tito-wrapper to deal with the changed versioning.

Unified package names to correspond to the dir names here.

Mar 23 '23 09:03 avshiliaev

Closing in favor of https://github.com/uyuni-project/uyuni/pull/6910

Apr 24 '23 08:04 aaannz

uyuni uyuni copied to clipboard

Automate changelog handling for container images

What does this PR change?

GUI diff

Documentation

Test coverage

Links

Changelogs

Re-run a test

uyuni
uyuni copied to clipboard