machine: Add workaround for Tumbleweed ssh_pwauth regression

[martinpitt]

See https://bugzilla.suse.com/show_bug.cgi?id=1237764

There is an upstream fix, but it will take a while to get to downstream. In the meantime, apply the fix locally.

Fixes #7488

• [x] image-refresh opensuse-tumbleweed

[cockpituous]

Failed. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-opensuse-tumbleweed-75fe837b-20250328-135647/log.html

[martinpitt]

@SludgeGirl Well, it got much further with the hack!

(22/23) Installing: mock-5.5-2.155.noarch [..done]
[...]
+ su builder -c 'sudo mock --verbose -i [...]
Traceback (most recent call last):
  File "/usr/bin/mock", line 64, in <module>
    from mockbuild import config
ModuleNotFoundError: No module named 'mockbuild'

That feels like a bug in mock? Missing some python dependency? I suppose we can add that manually somehow.

[SludgeGirl]

@martinpitt Oh huh, I'll poke it now

[martinpitt]

I did a local ./image-create -sv opensuse-tumbleweed and now are at the failing point where calling mock fails with the ImportError. I did a zypper search mock|grep python and there are a lot of unrelated matches, but I don't see something related. Also, rpm -ql mock already contains /usr/lib/python3.11/site-packages/mockbuild.

So it looks like the issue is that python3 is 3.13.2 now, so mock apparently wasn't rebuilt against 3.12 and 3.13? mock comes from a third-party OBS source obs://build.opensuse.org/system:packagemanager , so maybe that's the wrong one now?

I'm afraid that's where my knowledge ends, @SludgeGirl . Any idea?

[martinpitt]

Possible hack:

ln -s /usr/lib/python3.11/site-packages/mockbuild /usr/lib/python3.13/site-packages/mockbuild

but of course it would be better to get a matching build.

[SludgeGirl]

@martinpitt I managed to go a nab a maintainer role on that package, give it another rebuild now and it should work. It's still a couple versions behind but I can sort that next week

[cockpituous]

image-refresh opensuse-tumbleweed done: https://github.com/cockpit-project/bots/commits/image-refresh-opensuse-tumbleweed-20250331-060527

[cockpituous]

Success. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-opensuse-tumbleweed-75fe837b-20250331-054721/log.html

[martinpitt]

Thanks @SludgeGirl , mock worked again :tada: Let's see how the image refresh holds up in testing.

[martinpitt]

Hmm, the generated mock is still broken:

ERROR: builddep command missing.
Please install package dnf-plugins-core.

However, when I actually try to do this:

# su builder -c "sudo mock --install dnf-plugins-core"

Error: 
 Problem: package python3-dnf-plugins-core-4.4.4-2.4.noarch from opensuse-tumbleweed-oss requires python3-dnf >= 4.11.0, but none of the providers can be installed
  - package python3-dnf-4.23.0-1.1.noarch from opensuse-tumbleweed-oss conflicts with python3-dnf-plugins-core < 4.7.0 provided by python3-dnf-plugins-core-4.4.4-2.4.noarch from opensuse-tumbleweed-oss
  - package dnf-plugins-core-4.4.4-2.4.noarch from opensuse-tumbleweed-oss requires python3-dnf-plugins-core = 4.4.4-2.4, but none of the providers can be installed
  - conflicting requests

The generated mock doesn't have dnf installed at all, and I think that's by design (at least Fedora/RHEL don't do that, they use dnf from outside with a buildroot option). However, the host also doesn't have dnf installed, as that's using zypper.

The current main image (i.e. the very old one) still has some dnf packages installed in the host:

dnf-data-4.18.0-6.2.noarch
libdnf2-0.73.3-1.2.x86_64
python3-libdnf-0.73.3-1.2.x86_64
python3-dnf-4.18.0-6.2.noarch

but no dnf command. But there's also no dnf inside the mock. So somehow mock lost its ability to build source packages.

[SludgeGirl]

Mock itself is definitely working, this resolves successfully:

# su builder -c "sudo mock --install cockpit"

I'm digging a bit further into things now, I think either there's an issue with the tumbleweed-dnf container itself or somethings up with our current packaging for dnf

[SludgeGirl]

Okay looking a bit deeper dnf-plugins-core hadn't been updated in 6 months but dnf had been updated quite regularly. There's a sr open currently to update dnf-plugins-core to the latest version here: https://build.opensuse.org/request/show/1265156 I'm hoping it'll land early this week and it should fix the issues we're getting

Sorry for all the delays!

[martinpitt]

Let's give this another try, the above issue looks fixed now.

[cockpituous]

Failed. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-opensuse-tumbleweed-5e344e72-20250425-092823/log.html

[martinpitt]

@SludgeGirl I'm afraid mock is broken again :cry:

[SludgeGirl]

@martinpitt It looks like the location of the %suse_version macro has changed so build-deps.sh is failing to run, /usr/lib/rpm/suse/macros -> /usr/lib/rpm/macros.d/macros.susedist. I don't suppose you know what the reason is for fetching the it directly instead of letting rpm eval it through rpm --eval "%suse_version"? It feels like it'd be a little more robust

So this set of changes get us pretty far:

diff --git a/images/scripts/lib/build-deps.sh b/images/scripts/lib/build-deps.sh
index 31542b6d..501429c6 100755
--- a/images/scripts/lib/build-deps.sh
+++ b/images/scripts/lib/build-deps.sh
@@ -20,7 +20,7 @@ case "$OS_VER" in
     *suse*)
         # macro for determining suse version is %suse_version
         spec=$($GET "$COCKPIT_GIT/main/tools/cockpit.spec")
-        OS_VER_NO_VARIANT="suse_version $(awk '/%suse_version/{ print $2 }' /usr/lib/rpm/suse/macros)"
+        OS_VER_NO_VARIANT="suse_version $(rpm --eval '%suse_version')"
         ;;
     *)
         spec=$($GET "$COCKPIT_GIT/main/tools/cockpit.spec")
diff --git a/images/scripts/opensuse-tumbleweed.setup b/images/scripts/opensuse-tumbleweed.setup
index bfd5f741..38f2c45b 100755
--- a/images/scripts/opensuse-tumbleweed.setup
+++ b/images/scripts/opensuse-tumbleweed.setup
@@ -135,10 +135,11 @@ zypper install -y cockpit
 
 zypper addrepo https://download.opensuse.org/repositories/system:packagemanager/openSUSE_Tumbleweed/system:packagemanager.repo
 zypper --non-interactive --gpg-auto-import-keys  refresh
-zypper install -y mock mock-core-configs sudo
+zypper install -y mock mock-core-configs sudo dnf-plugins-core
 
 echo "%wheel ALL=(ALL) ALL" > /etc/sudoers.d/90-cockpit-wheel
 echo "builder ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-susebuild
+echo "config_opts['use_bootstrap'] = False" >>/etc/mock/site-defaults.cfg
 useradd -c Builder -G mock builder
 
 su builder -c "sudo mock --verbose -i $(/var/lib/testvm/build-deps.sh "${ID} ${VERSION_ID}")"

With those it fully builds the image, just when we come to prepare it I get a pytest issue:

DEBUG: ++ '[' -f _current_flavor ']'
DEBUG: ++ cat _current_flavor
DEBUG: + last_flavor=python313
DEBUG: + '[' -z python313 ']'
DEBUG: + '[' python313 '!=' python311 ']'
DEBUG: + '[' -d build ']'
DEBUG: + mv build _build.python313
DEBUG: + '[' -d _build.python311 ']'
DEBUG: + mv _build.python311 build
DEBUG: + echo python311
DEBUG: + python_flavor=python311
DEBUG: + PYTHONPATH='/builddir/build/BUILD/cockpit-337.dev2+g563d0f624-build/BUILDROOT!!_{/usr/bin/python3.11}_not_installed_!!'
DEBUG: + PYTHONDONTWRITEBYTECODE=1
DEBUG: + 'pytest-!!_{/usr/bin/python3.11}_not_installed_!!' --ignore=_build.python311 --ignore=_build.python312 --ignore=_build.python313 -v
DEBUG: /var/tmp/rpm-tmp.0Z3d2x: line 158: pytest-!!_{/usr/bin/python3.11}_not_installed_!!: No such file or directory
DEBUG: error:
DEBUG: RPM build errors:
DEBUG: Bad exit status from /var/tmp/rpm-tmp.0Z3d2x (%check)
DEBUG:     Bad exit status from /var/tmp/rpm-tmp.0Z3d2x (%check)
DEBUG: Child return code was: 1

[martinpitt]

@SludgeGirl Nice, thanks!

I don't suppose you know what the reason is for fetching the it directly instead of letting rpm eval

Not really I'm afraid. This was part of the initial commit 26b51601c43392ec00f2514a2da235edb5f22d31 by @Lunarequest . But your's looks plausible.

I pushed your changes (please cross-check the commit message, as I'm putting words into your mouth). Let's get a current build log for the Python issue.

[cockpituous]

image-refresh opensuse-tumbleweed done: https://github.com/cockpit-project/bots/commits/image-refresh-opensuse-tumbleweed-20250425-140511

[cockpituous]

Success. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-opensuse-tumbleweed-9a35d170-20250425-135039/log.html

[SludgeGirl]

I pushed your changes (please cross-check the commit message, as I'm putting words into your mouth). Let's get a current build log for the Python issue.

Yeah I think that's a fine enough commit message

[martinpitt]

So surprisingly the refresh worked, but there are multiple OS regressions in machines: at least a qemu crash, and an ImportError in targetcli (test is still running). This image refresh changes a ton of packages (as the previous image refresh was more than two months ago), so this will need some more work in tumbleweed I'm afraid :cry: Alternatively we could also mark all these tests as known failures, but hopefully at least the readline python import has a simple workaround? (maybe just a missing dependency?)

[SludgeGirl]

Yeah we just need to add in

diff --git a/images/scripts/opensuse-tumbleweed.setup b/images/scripts/opensuse-tumbleweed.setup
index 38f2c45b..4a5db8a1 100755
--- a/images/scripts/opensuse-tumbleweed.setup
+++ b/images/scripts/opensuse-tumbleweed.setup
@@ -86,6 +86,7 @@ podman \
 valkey \
 socat \
 strace \
+python313-targetcli-fb \
 targetcli \
 tcsh \
 bzip2 \

targetcli by default wasn't to run against 3.11, if we install python313-targetcli-fb with it, it'll resolve correctly to 3.13

[SludgeGirl]

This image refresh changes a ton of packages (as the previous image refresh was more than two months ago), so this will need some more work in tumbleweed I'm afraid 😢

It was bound to be the case. I would say lets mark them as known failures for now, I've got quite a bit I'm handling at the minute but the tests are a fairly high priority for us. We'll handle them as soon as we can

CC: @Lunarequest @Nykseli

[cockpituous]

image-refresh opensuse-tumbleweed done: https://github.com/cockpit-project/bots/commits/image-refresh-opensuse-tumbleweed-20250427-030837

[cockpituous]

Success. Log: https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-opensuse-tumbleweed-ebca2e0a-20250427-025241/log.html

[martinpitt]

@SludgeGirl Thanks! I applied the targetcli hack. machines tests look much better now. libvirt was updated to 11.2.0, thus are affected by #7648 now. I'll copy the naughty.

We can create a naughty for testAddDiskCustomPath, but we normally require a corresponding downstream bug report.

The testExternalConsoleMultiHost authentication failure needs checking. I'll do that when it's not the middle of the night.

[martinpitt]

The testExternalConsoleMultiHost authentication failure failure happens due to a crash in beiboot with calling tempfile.TemporaryDirectory().

FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']

Trying to create a file in /tmp or /var/tmp in beiboot.py explains why:

OSError: [Errno 30] Read-only file system: '/var/tmp/log'

Or more directly:

# nsenter -t $(pgrep cockpit-ws) -m touch /tmp/foo
touch: cannot touch '/tmp/foo': Read-only file system

But there is something really fishy here: /usr/libexec/cockpit-ws runs as root (it's really not supposed to, and dangerous), and /usr/lib/systemd/system/cockpit-wsinstance-http.service has DynamicUser=no (it's meant to be yes). It seems opensuse patches the cockpit units? That makes /tmp read-only. I changed /usr/lib/systemd/system/cockpit-wsinstance-http.service a bit and bisected to ProtectSystem=strict. This doesn't really without also setting PrivateTmp=yes. When I add that to the unit, the test works fine.

@Lunarequest @SludgeGirl I don't know where the opensuse packaging of cockpit lives, but I propose we'll look together at your unit patches. You need to at least add PrivateTmp=yes, but preferably let's see which changes we can revert, and why DynamicUser was switched off.

[martinpitt]

I created known issues #7699 and #7700, so that this should finally get green. We need to push this through to catch up with reality. The QEMU crash is "meh", but the broken cockpit units are important.

[jelly]

In my memory we tested opensuse on cockpit-podman but that ain't the case so all good :+1: