virtiofsd not in path breaks podman start

[baude]

Issue Description

It seems like the recent addition of virtiofsd broke podman start on fedora. The virtiofsd executable sits in /usr/libexec and is not present in path.

[baude@baudework ~]$ podman machine start
Starting machine "podman-machine-default"
ERRO[0000] process 6160 has not ended                   
Error: failed to find virtiofsd: exec: "virtiofsd": executable file not found in $PATH

Steps to reproduce the issue

Steps to reproduce the issue 1. 2. 3.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

https://github.com/containers/podman/pull/22920#discussion_r1630919819

In general if you want to hard code paths you can add this back but this is horrible as it doesn't scale across distros.

One thing that we should change though is not to lookup in $PATH. This doesn't make sense and the returned error to users is confusing.

[afbjorklund]

Seems to be missing from the build-from-source documentation, but then again so is gvproxy etc.

• https://github.com/containers/podman.io/issues/285

Upstream isn't clear on where to install:

https://gitlab.com/virtio-fs/virtiofsd

So normally it just ends up in /usr/local/bin.

But /usr/local/libexec is missing from the list...

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[afbjorklund]

With v5.2.0, it will now find /usr/local/libexec/podman/virtiofsd

So it can be installed next to /usr/local/libexec/podman/gvproxy*

• https://packages.ubuntu.com/noble/virtiofsd uses /usr/libexec too

* For Fedora (only), gvisor-tap-proxy does not need that helper symlink...

Since it it installs gvproxy in /usr/libexec/podman instead of /usr/bin.

• https://github.com/containers/podman/issues/11665

---

QEMU 8.0 removed the bundled virtiofsd tool

So it needs to be installed separately anyway.

Note: Podman requires upgrading to QEMU 7.2.0

656bf7b76447b2b14eb73be4497ef407180cf45b

[rhatdan]

Should this issue be closed?

[afbjorklund]

The current hack will work for latest Fedora (40) and latest Ubuntu (24.04). Others will have to work around the issue with symlinks from the helper dir...

https://github.com/containers/common/commit/d2e005fa21e84d01cad8ea514b0e26e2a859d11d

/usr/local/libexec/podman/gvproxy -> /usr/local/bin/gvproxy

/usr/local/libexec/podman/virtiofsd -> /usr/local/bin/virtiofsd

---

Or wherever those dependencies are installed, like lib or libexec?

They should probably also be added to the machine documentation.

They are not needed when running podman locally, only podman machine.

But it could be a point to build them with podman anyway, "just in case".

like catatonit ? (if installed system-wide, and not just for podman)

/usr/local/libexec/podman/catatonit -> /usr/local/bin/catatonit

[afbjorklund]

Upstream QEMU has a JSON file (vhost-user.json), where the path is defined.

But it (vhost-user config) might have gotten lost, in the Rust Rewrite?

It also needs to be checked, that it is not the old version (QEMU < 8.0)

Lima has a workaround to call --version, which only the new implements...

[afbjorklund]

They are not needed when running podman locally, only podman machine.

But it could be a point to build them with podman anyway, "just in case".

If I understand packaging rules, it's supposed to go in a subpackage since 5.2:

• https://github.com/containers/podman/pull/23095

https://packages.fedoraproject.org/pkgs/podman/podman-machine/

So "podman-machine" is where gvproxy/virtiofsd and even the new qemu lives.

[mryo0826]

The version of podman managed by Fedora 40's dnf has been updated to v5.2.0. It probably happened today (2024/08/12). This version causes the bug in this issue. The problem did not occur with v5.1.2 until yesterday. It seems to be happening because qume installed with dnf dependencies is 8.2.2. I specified podman v5.0.0 in dnf and installed it, but since qume is still 8.2.2, this problem occurs.

[afbjorklund]

You need to install the "podman-machine" package, once it is available in Fedora:

https://packages.fedoraproject.org/pkgs/podman/podman-machine/ --> https://packages.fedoraproject.org/pkgs/virtiofsd/virtiofsd/

Not sure what happened to the GA (it only has RC), but that's a Fedora question.

A workaround is to create the /usr/libexec/podman/virtiofsd symlink. And to make sure the package is installed, ~~it seems missing from fc40~~.

EDIT: just a bug in the web page, it does have Requires: virtiofsd

[mryo0826]

You need to install the "podman-machine" package, once it is available in Fedora:

Sorry I don't understand what you posted before. I installed the "podman-machine" package.

But now I'm getting another error.

$ podman machine start
Starting machine "podman-machine-default"
Error: qemu exited unexpectedly with exit code 1, stderr: Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory

I have confirmed that "qemu-kvm" is installed in my environment.

Are there still other required modules that have different paths that are causing the error?

[afbjorklund]

There might be other steps needed too, to set up KVM

Ubuntu has a script called kvm-ok, to check CPU setup

---

It doesn't have anything to do with virtiofsd, though

https://docs.fedoraproject.org/en-US/quick-docs/virtualization-getting-started/#_qemukvm_without_libvirt

[cgwalters]

I think we need a public service announcement of the form:

"Are you using podman-machine on Linux (e.g. Fedora?); then dnf -y install podman-machine or equivalent"

Hmm...you know while I tend to agree with Paul about not hardcoding file paths, on the flip side it would be very practical to add a downstream patch (or maybe an upstream variant) that basically checks on podman machine start if /usr/libexec/podman/virtiofsd exists, and if it doesn't tells you to install that package.

@baude I am going to go ahead and proactively close this issue as "fixed" with the resolution "please install podman-machine", but of course feel free to reopen if you disagree!

[afbjorklund]

The same issue already existed for gvproxy, but better documentation or error messages could help...

I think it is deliberately not mentioned in Podman and Podman Desktop, so that people avoid the machine VM

---

But I could add "virtiofsd" to the notes I was keeping for v4.9, to make the instructions valid for v5.2:

• https://github.com/containers/podman-desktop/discussions/5762

[cgwalters]

(OK I know this is a closed issue, but we can gather some quick consensus here before making a new one (or straight to a PR))

How about this: podman-machine on Linux requires /usr/libexec/podman/machine to exist (as a symlink to ../../bin/podman) - and the presence of that file signals that all dependencies of podman-machine are installed (and it's shipped by the podman-machine package on Fedora etc.). If not, podman has a generic message that tells you to go install it using your OS/distro tooling. We could add a build tag that allows customizing the error message.

[afbjorklund]

Or... it just says "file not found"?

When packaged, it will have pulled in the dependencies for you but otherwise I think putting it in the documentation is "good enough"

[cgwalters]

cc @baude for opinions on :arrow_up

[Luap99]

podman-machine on Linux requires /usr/libexec/podman/machine

That sounds like extra work for no benefit. And AFAIK other distros do not use a libexec dir at all so hard coding this does not work for them. Neither will it work for users installing binaries in their home dir. IMO if we do not find qemu/gvproxy/virtiofsd,etc.. we should just throw better errors in general and simply document what packages are required on what distros.

[germag]

Sorry, I'm late to this

Since /usr/libexec is not in the PATH it will fail even if virtiofsd is installed, adding HELPER_BINARIES_DIR="/usr/libexec" in the build step of the rpm spec (it will be added to additionalHelperBinariesDir at link time) will fix the issue, without interfering with other distributions

[Luap99]

Since /usr/libexec is not in the PATH it will fail even if virtiofsd is installed, adding HELPER_BINARIES_DIR="/usr/libexec" in the build step of the rpm spec (it will be added to additionalHelperBinariesDir at link time) will fix the issue, without interfering with other distributions

You should install podman-machine which adds a symlink from the podman default helper dir to virtiofsd.

[germag]

Since /usr/libexec is not in the PATH it will fail even if virtiofsd is installed, adding HELPER_BINARIES_DIR="/usr/libexec" in the build step of the rpm spec (it will be added to additionalHelperBinariesDir at link time) will fix the issue, without interfering with other distributions

You should install podman-machine which adds a symlink from the podman default helper dir to virtiofsd.

then I'll add podman-machine as dependency in the podman-bootc rpm spec, thanks!