Start sshd service failed on fedora 36 iot system(installed by iot-commit)

[henrywang]

Describe the bug Run https://github.com/osbuild/osbuild-composer/blob/main/test/cases/ostree.sh on Fedora 36 VM. The installed iot vm starts sshd service failed with error:

May 30 07:15:47 localhost.localdomain sshd[1115]: Privilege separation user sshd does not exist
May 30 07:15:47 localhost.localdomain systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION
May 30 07:15:47 localhost.localdomain systemd[1]: sshd.service: Failed with result 'exit-code'.
May 30 07:15:47 localhost.localdomain systemd[1]: Failed to start sshd.service - OpenSSH server daemon.

Environment

• OS version (/etc/os-release and /etc/redhat-release): Fedora 36
• osbuild-composer version (rpm -qi osbuild-composer): osbuild-composer-51-1.fc36.x86_64

To Reproduce Steps to reproduce the behavior:

• Deploy a Fedora 36 vm on PSI openstack.
• Run https://github.com/osbuild/osbuild-composer/blob/main/test/cases/ostree.sh

Expected behavior Run ostree.sh without error.

Additional context Can't ssh to iot system which is installed by iot-commit. Console into iot system, found the sshd servier failed to start.

[achilleas-k]

There's something wrong with our ostree deployments on f36. I don't know the root cause yet, but it's misbehaving in a few ways. One part is the merging of passwd and group entries in the deployment, which is what is causing this issue, but also the rpm db isn't properly set up either.

[gicmo]

So this is because nssaltfiles is not configured properly, although the package is installed. This is due to the detection logic in rpm-ostree and authselect on Fedora 36 taking over nssswitch configuration. It has been worked around in the upstream authselect config file via 0010-spec-fix-detection-of-ostree-system.patch, but we don't get that fix, because we don't set /run/ostree-booted.

[gicmo]

I have a draft PR that I confirmed fixes the issue at https://github.com/osbuild/osbuild/pull/1085

[achilleas-k]

This should be fixed, but I'd like to keep it open until we enable testing in CI for it again.