Error: machine did not transition into running state: ssh error: machine not in running state

[szarykott]

Issue Description

I am trying to setup podman on Windows 10, but to no avail. Every time I try to start podman machine I receive an error that says:

Error: machine did not transition into running state: ssh error: machine not in running state

See log below for more details.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine init --rootful
2. podman machine start

Describe the results you received

podman --log-level=trace machine start
time="2024-04-02T10:06:48+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level trace"
time="2024-04-02T10:06:48+02:00" level=debug msg="Using Podman machine with `wsl` virtualization provider"
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RadosławKot\\.config\\containers\\podman\\machine\\wsl is 59"
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RadosławKot\\.local\\share\\containers\\podman\\machine\\wsl is 64"
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RadosławKot\\.local\\share\\containers\\podman\\machine\\wsl\\cache is 70"
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RADOSA~1\\AppData\\Local\\Temp\\podman is 43"
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RadosławKot\\.config\\containers\\podman\\machine\\wsl\\podman-machine-default.json is 87"
Starting machine "podman-machine-default"
time="2024-04-02T10:06:48+02:00" level=debug msg="writing configuration file \"C:\\\\Users\\\\RadosławKot\\\\.config\\\\containers\\\\podman\\\\machine\\\\wsl\\\\podman-machine-default.json\""
time="2024-04-02T10:06:48+02:00" level=debug msg="socket length for C:\\Users\\RADOSA~1\\AppData\\Local\\Temp\\podman\\gvproxy.pid is 55"
time="2024-04-02T10:06:48+02:00" level=debug msg="Running command: C:\\Users\\RadosławKot\\AppData\\Local\\Microsoft\\WindowsApps\\wsl.exe [-u root -d podman-machine-default /root/bootstrap]"
API forwarding for Docker API clients is not available due to the following startup failures.
        could not start api proxy since expected pipe is not available: podman-machine-default

Podman clients are still able to connect.
time="2024-04-02T10:07:09+02:00" level=debug msg="writing configuration file \"C:\\\\Users\\\\RadosławKot\\\\.config\\\\containers\\\\podman\\\\machine\\\\wsl\\\\podman-machine-default.json\""
Error: machine did not transition into running state: ssh error: machine not in running state
time="2024-04-02T10:07:09+02:00" level=debug msg="Shutting down engines"

Describe the results you expected

Machine should start successfully.

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.20240313132120223048.main.19.gaffab49.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.96
    systemPercent: 0.02
    userPercent: 0.01
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: HJK82J3-JLABS
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.146.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 15909048320
  memTotal: 16626651136
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240329131657512331.main.31.g2c315a1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240329131649297909.main.62.gad066d4.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240331210158645743.main.18.gd05a5dd.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 22fa748a7dd029cd597fc96a2afb78b84cf819e6
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 32m 48.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 599588864
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.1.0-dev-e1f49529f
  Built: 1711929600
  BuiltTime: Mon Apr  1 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.0-dev-e1f49529f

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

• Host OS: Windows 10, WSL2
• Docker is not installed
• removing and initializing machine again does not help
• removing and reinstalling Podman does not help
• removing and reinstalling entire WSL does not help
• issue was present in older versions of Podman as well

wsl -l -v output

  NAME                      STATE           VERSION
* Ubuntu                    Stopped         2
  podman-machine-default    Running         2
  Ubuntu-20.04              Stopped         2

Additional information

Issue happens every time I issue the command.

[baude]

odd, the fact that your podman info command reports all of that suggests that it is working. Did you use podman to connect to other hosts or vms at one point? what do you report for podman system connection ls?

[szarykott]

Did you use podman to connect to other hosts or vms at one point?

No, it never worked

what do you report for podman system connection ls?

podman system connection ls
Name                         URI                                                          Identity                                                             Default     ReadWrite
podman-machine-default       ssh://[email protected]:50900/run/user/1000/podman/podman.sock  C:\Users\RadosławKot\.local\share\containers\podman\machine\machine  false       true
podman-machine-default-root  ssh://[email protected]:50900/run/podman/podman.sock            C:\Users\RadosławKot\.local\share\containers\podman\machine\machine  true        true

[baude]

can i ask, were did you get podman from ? i assume you downloaded the installer from our releases on github ?

[szarykott]

Yes, it was from https://github.com/containers/podman/releases, specifically podman-5.0.1-setup.exe

[Luap99]

Did v5.0.0 worked (i.e. is this a regression)?

[n1hility]

could not start api proxy since expected pipe is not available: podman-machine-default

^ this suggests you might have a lingering win-sshproxy process

Could you try the following:

1. podman machine stop
2. wsl --shutdown
3. taskkill /f /im win-sshproxy.exe

[szarykott]

Did v5.0.0 worked (i.e. is this a regression)?

No, I recall having this issue with 4.9.3 as well (and 5.0.0).

^ this suggests you might have a lingering win-sshproxy process Could you try the following: podman machine stop wsl --shutdown taskkill /f /im win-sshproxy.exe

Tried it, it did not help

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[szarykott]

Any ideas? I upgrade to new version of Podman once it arrives, but issue still remains

[zhangchaoza]

I have the same error, and I just use the old machine-os in manual when init a machine.

Download the release 👉 the lastest release can be inited sucessed I tested

podman machine init --image C:\Users\zhangchao\Downloads\5.1-rootfs-amd64.tar.zst

[szarykott]

IT WORKED! Thank you @zhangchaoza!

Now question remains - why? I can use outdated image forever, if something can be fixed, it should.

[Luap99]

Do you use networkingMode=mirrored in WSL?

[timfel]

Just to add my 2c, got the same error, and indeed had networkingMode=mirrored in ~\.wslconfig. Setting it back to nat solved it.

[szarykott]

Ticket can be closed, since I got a new PC in the meantime and by accident @zhangchaoza 's advice solved the issue