`podman machine start` fails on macOS EC2 instance on AWS

[seanhelm]

Issue Description

I am trying to set up podman on both mac1.metal (x64) and mac2.metal (ARM) EC2 instances in AWS. In both cases after installing and initializing the VM, the podman machine start --log-level=debug command fails with:

Error: dial unix /tmp/podman/qmp_podman-machine-default.sock: connect: no such file or directory

I found a similar, unresolved issue relating to this: https://github.com/containers/podman/issues/25693

Steps to reproduce the issue

Steps to reproduce the issue

1. Download the package installer: curl -L -o /tmp/podman.pkg https://github.com/containers/podman/releases/download/v4.5.1/podman-installer-macos-amd64.pkg
2. Install it on the machine: sudo installer -pkg /tmp/podman.pkg -target /
3. Change the owner of /Users/ec2-user/.ssh directory to ec2-user (it's initially owned by root on EC2)
4. Run podman machine init as user ec2-user
5. Run podman machine start --log-level=debug as user ec2-user

Describe the results you received

The machine successfully initialized, but failed to start when running podman machine start --log-level=debug (on mac2.metal ARM):

time="2025-03-27T21:08:02Z" level=info msg="podman filtering at log level debug"
	Starting machine "podman-machine-default"
	[/opt/podman/qemu/bin/gvproxy -listen-qemu unix:///tmp/podman/qmp_podman-machine-default.sock -pid-file /tmp/podman/podman-machine-default_proxy.pid -ssh-port 49547 -forward-sock /Users/ec2-user/.local/share/containers/podman/machine/qemu/podman.sock -forward-dest /run/user/501/podman/podman.sock -forward-user core -forward-identity /Users/ec2-user/.ssh/podman-machine-default --debug]
	Error: dial unix /tmp/podman/qmp_podman-machine-default.sock: connect: no such file or directory
	time="2025-03-27T21:08:05Z" level=debug msg="Shutting down engines"

Describe the results you expected

After running podman machine start --log-level=debug, I expected the VM to successfully start. I would then be able to run commands such as podman ps, etc.

podman info output

Output: Error: failed to connect: dial tcp 127.0.0.1:49547: connect: connection refused

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[ninja-quokka]

Hi @seanhelm, thank you for the bug report!

I would like to first ask if you have tried reproducing this issue with the latest Podman release? I note from your issue report you are using v4.5.1 which as this point is almost 2 years old.

For reference, here is the current, latest release: https://github.com/containers/podman/releases/tag/v5.4.1

[seanhelm]

Hi @ninja-quokka, thanks for following up. I retried it with v5.4.1, getting a similar but slightly different error when calling podman machine start --log-level=debug on m1.metal:

time="2025-03-28T13:45:31Z" level=info msg="podman filtering at log level debug"
	time="2025-03-28T13:45:31Z" level=debug msg="Using Podman machine with `applehv` virtualization provider"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.config/containers/podman/machine/applehv is 57"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv is 62"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv/cache is 68"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman is 11"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.config/containers/podman/machine/applehv/podman-machine-default.json is 85"
	Starting machine "podman-machine-default"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.config/containers/podman/machine/applehv is 57"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv is 62"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv/cache is 68"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman is 11"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.config/containers/podman/machine/applehv/podman-machine-default.json is 85"
	time="2025-03-28T13:45:31Z" level=debug msg="connection refused: http://localhost:49294/vm/state"
	time="2025-03-28T13:45:31Z" level=debug msg="writing configuration file \"/Users/ec2-user/.config/containers/podman/machine/applehv/podman-machine-default.json\""
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman/gvproxy.pid is 23"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.config/containers/podman/machine/applehv is 57"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv is 62"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /Users/ec2-user/.local/share/containers/podman/machine/applehv/cache is 68"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman is 11"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman/podman-machine-default-api.sock is 43"
	time="2025-03-28T13:45:31Z" level=debug msg="{true 1500 map[forward-dest:[/run/user/501/podman/podman.sock] forward-identity:[/Users/ec2-user/.local/share/containers/podman/machine/machine] forward-sock:[/tmp/podman/podman-machine-default-api.sock] forward-user:[core]] [] [] map[] /tmp/podman/gvproxy.log /tmp/podman/gvproxy.pid 49286}"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman/podman-machine-default-gvproxy.sock is 47"
	time="2025-03-28T13:45:31Z" level=debug msg="gvproxy command-line: /opt/podman/bin/gvproxy -debug -mtu 1500 -ssh-port 49286 -listen-vfkit unixgram:///tmp/podman/podman-machine-default-gvproxy.sock -forward-sock /tmp/podman/podman-machine-default-api.sock -forward-dest /run/user/501/podman/podman.sock -forward-user core -forward-identity /Users/ec2-user/.local/share/containers/podman/machine/machine -pid-file /tmp/podman/gvproxy.pid -log-file /tmp/podman/gvproxy.log"
	time="2025-03-28T13:45:31Z" level=debug msg="socket length for /tmp/podman/podman-machine-default-gvproxy.sock is 47"
	time="2025-03-28T13:45:31Z" level=debug msg="checking that \"gvproxy\" socket is ready"
	time="2025-03-28T13:46:02Z" level=debug msg="writing configuration file \"/Users/ec2-user/.config/containers/podman/machine/applehv/podman-machine-default.json\""
	Error: unable to connect to "gvproxy" socket at "/tmp/podman/podman-machine-default-gvproxy.sock"
	time="2025-03-28T13:46:03Z" level=debug msg="Shutting down engines"

[baude]

hello @seanhelm normally that gvproxy socket ends up somewhere like /var/folders/s1/q54t24qd36q_ng7gzw7mcbsm0000gn/T/podman/podman-machine-default-gvproxy.sock so there must be some environmental overrides going on.

Also, with apple stuff, we always ask where you got the podman binaries and supporting applications, like gvisor and vfkit from. If it came from brew, please double triple check all the binaries are of the correct architecture.

[github-actions[bot]]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[seanhelm]

I was able to resolve this issue. Originally, I was directly invoking /opt/podman/bin/podman machine init and /opt/podman/bin/podman machine start. I suspect that because /opt/podman/bin wasn't in my $PATH, it was having issues with gvproxy.

Adding /opt/podman/bin to the PATH and then running podman machine init and podman machine start works.