podman icon indicating copy to clipboard operation
podman copied to clipboard
verified publisher icon containers

machine: timeout in start

Open edsantiago opened this issue 1 year ago • 7 comments

Seeing this very often - much more than the table below shows, because my flake logger only logs once a PR is merged.

           Starting machine "blah blah"

[+1207s]   [FAILED] Timed out after 600.001s.
           Expected process to exit.  It did not.
  • darwin : machine-mac podman darwin rootless host sqlite
    • PR #22389
      • 04-16 07:22 in podman machine rm Remove machine sharing ssh key with another machine
  • fedora-39-aarch64 : machine-linux podman fedora-39-aarch64 rootless host sqlite
    • PR #22545
      • 04-30 07:46 in podman machine start start two machines in parallel
      • 04-30 06:08 in podman machine init machine init with volume
    • PR #22346
x x x x x x
machine-linux(4) podman(5) fedora-39-aarch64(4) rootless(5) host(5) sqlite(5)
machine-mac(1) darwin(1)

edsantiago avatar Apr 30 '24 16:04 edsantiago

  • darwin : machine-mac podman darwin rootless host sqlite
    • PR #22389
      • 04-16 07:22 in podman machine rm Remove machine sharing ssh key with another machine
  • fedora-39-aarch64 : machine-linux podman fedora-39-aarch64 rootless host sqlite
    • PR #22584
      • 05-03 06:43 in podman machine set set while machine already running
    • PR #22582
      • 05-06 09:01 in podman machine rm Remove machine sharing ssh key with another machine
      • 05-06 08:23 in run basic podman commands Volume ops
      • 05-02 20:21 in podman machine init machine init rootless docker.sock check
    • PR #22581
      • 05-02 17:38 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 05-02 17:03 in podman machine start start only starts specified machine
      • 05-02 17:03 in podman machine init machine init rootless docker.sock check
      • 05-02 15:58 in podman machine init machine init rootful with docker.sock check
    • PR #22552
      • 04-30 17:09 in podman machine set set rootful with docker sock change
      • 04-30 16:39 in podman machine list list machine: check if running while starting
      • 04-30 15:53 in podman machine reset reset with running machine and other machines idle
    • PR #22548
      • 04-30 15:10 in podman machine rm Remove machine sharing ssh key with another machine
      • 04-30 14:40 in podman machine ssh ssh to running machine and check os-type
      • 04-30 13:26 in run basic podman commands Volume ops
      • 04-30 12:50 in podman machine start start machine already started
      • 04-30 12:18 in podman machine reset reset with running machine and other machines idle
    • PR #22545
      • 04-30 07:46 in podman machine start start two machines in parallel
      • 04-30 06:08 in podman machine init machine init with volume
    • PR #22346
  • windows : machine-hyperv podman windows rootless host sqlite
    • PR #22608
      • 05-06 07:26 in podman machine start C:/Users/Administrator/AppData/Local/cirrus-ci-build/repo/pkg/machine/e2e/start_test.go:13 start simple machine C:/Users/Administrator/AppData/Local/cirrus-ci-build/repo/pkg/machine/e2e/start_test.go:25
x x x x x x
machine-linux(20) podman(22) fedora-39-aarch64(20) rootless(22) host(22) sqlite(22)
machine-hyperv(1) darwin(1)
machine-mac(1) windows(1)

edsantiago avatar May 06 '24 15:05 edsantiago

Another Windows one (on main): https://api.cirrus-ci.com/v1/artifact/task/6723119224717312/html/machine-linux-podman-fedora-39-aarch64-rootless-host-sqlite.log.html

cevich avatar May 08 '24 18:05 cevich

Different one: here the test fails, but not via timeout. machine wsl:

  C> podman.exe machine start 492b48671720
  Starting machine "492b48671720"
  your 131072x1 screen size is bogus. expect trouble

  This machine is currently configured in rootless mode. If your containers
  require root permissions (e.g. ports < 1024), or if you run into compatibility
  issues with non-podman clients, you can switch using the following command:

  	podman machine set --rootful 492b48671720

  API forwarding listening on: npipe:////./pipe/docker_engine

  Docker API clients default to this address. You do not need to set DOCKER_HOST.
  Error: machine did not transition into running state: ssh error: machine is not listening on ssh port

Is this the same bug? Shall I assign it to this issue?

edsantiago avatar May 13 '24 15:05 edsantiago

I seem to remember @baude was working/debugging on this or something related to "machine fails to start" a month or so ago. Dunno if it's the same thing or different, but in terms of getting it fixed, that's who I'd start with.

cevich avatar May 21 '24 15:05 cevich

A friendly reminder that this issue had no activity for 30 days.

github-actions[bot] avatar Jun 22 '24 00:06 github-actions[bot]

The past 30 days. It's tempting to focus on Mac because that's what's hitting us so hard right now in #23154 and #23157, but this is happening on windows and linux too.

x x x x x x
machine-mac(9) podman(17) darwin(9) rootless(17) host(17) sqlite(17)
machine-hyperv(4) fedora-40-aarch64(4)
machine-linux(4) windows(4)

edsantiago avatar Jul 02 '24 11:07 edsantiago

I've just spent ten minutes blindly assigning all podman machine flakes to this issue. I looked at logs for some of them, and, some of them include the timeout, some are different errors. I do not have the time nor interest in opening issues for every podman-machine failure, so the list below is not entirely accurate. Still, I hope it helps in some wah.

Podman machine CI is super broken right now. I hope this helps someone diagnose and fix it.

x x x x x x
machine-mac(82) podman(205) darwin(82) rootless(205) host(205) sqlite(205)
machine-linux(69) windows(54)
machine-hyperv(44) fedora-39-aarch64(37)
machine-wsl(10) fedora-40-aarch64(32)

edsantiago avatar Jul 22 '24 16:07 edsantiago

Executive decision: this issue is now a one-stop catchall for all podman-machine flakes. There are too many flakes, I don't have the time to look at each one, so I'm just doing an automatic lump of all flakes with "machine" in the test name into this issue. If anyone cares about podman-machine, please feel free to start tackling these.

Here's the last two weeks. Have fun.

  • darwin : machine-mac podman darwin rootless host sqlite
    • PR #23456
      • 07-31 16:40 in podman machine start start simple machine
      • 07-31 16:40 in podman machine start start machine already started
      • 07-31 16:40 in podman machine start start machine with conflict on SSH port
      • 07-31 16:40 in podman machine start start only starts specified machine
      • 07-31 16:40 in podman machine start start two machines in parallel
      • 07-31 16:40 in run basic podman commands Basic ops
      • 07-31 16:40 in run basic podman commands Volume ops
      • 07-31 16:40 in run basic podman commands Podman ops with port forwarding and gvproxy
      • 07-31 16:40 in run basic podman commands podman volume on non-standard path
      • 07-31 16:40 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 07-31 16:40 in podman machine info machine info
      • 07-31 16:40 in podman machine ssh ssh to non-running machine
      • 07-31 16:40 in podman machine ssh ssh to running machine and check os-type
      • 07-31 16:40 in podman machine init simple init
      • 07-31 16:40 in podman machine init simple init with start
      • 07-31 16:40 in podman machine init simple init with username
      • 07-31 16:40 in podman machine init machine init with cpus, disk size, memory, timezone
      • 07-31 16:40 in podman machine init machine init with volume
      • 07-31 16:40 in podman machine init machine init rootless docker.sock check
      • 07-31 16:40 in podman machine init machine init rootful with docker.sock check
      • 07-31 16:40 in podman machine init init should cleanup on failure
      • 07-31 16:40 in podman machine init verify a podman 4 config does not break podman 5
      • 07-31 16:40 in podman machine init machine init with rosetta=true
      • 07-31 16:40 in podman machine init machine init with rosetta=false
      • 07-31 16:40 in podman machine rm Remove machine
      • 07-31 16:40 in podman machine rm Remove running machine
      • 07-31 16:40 in podman machine rm machine rm --save-ignition --save-image
      • 07-31 16:40 in podman machine rm Remove machine sharing ssh key with another machine
      • 07-31 16:40 in podman machine rm Removing all machines doesn't delete ssh keys
      • 07-31 16:40 in podman machine list list machine
      • 07-31 16:40 in podman machine list list machines with quiet or noheading
      • 07-31 16:40 in podman machine list list machine: check if running while starting
      • 07-31 16:40 in podman machine list list with --format
      • 07-31 16:40 in podman machine list list machine in machine-readable byte format
      • 07-31 16:40 in podman machine list list machine in human-readable format
      • 07-31 16:40 in podman inspect stop inspect two machines
      • 07-31 16:40 in podman inspect stop inspect with go format
      • 07-31 16:40 in podman inspect stop inspect shows a unique socket name per machine
      • 07-31 16:40 in podman machine stop Stop running machine
      • 07-31 16:40 in podman machine reset reset machine with one defined machine
      • 07-31 16:40 in podman machine reset reset with running machine and other machines idle
    • PR #23275
      • 07-29 16:26 in [Cirrus] CI agent stopped responding!
  • fedora-40-aarch64 : machine-linux podman fedora-40-aarch64 rootless host sqlite
  • windows : machine-hyperv podman windows rootless host sqlite
    • PR #23466
    • PR #23462
    • PR #23425
      • 07-29 08:18 in 22551
      • 07-29 08:18 in 22551
      • 07-29 08:18 in podman machine stop Stop running machine
      • 07-29 08:18 in run basic podman commands Basic ops
      • 07-29 08:18 in run basic podman commands Volume should be disabled by command line
      • 07-29 08:18 in run basic podman commands Podman ops with port forwarding and gvproxy
      • 07-29 08:18 in run basic podman commands podman volume on non-standard path
      • 07-29 08:18 in podman machine init simple init with start
      • 07-29 08:18 in podman machine init machine init with cpus, disk size, memory, timezone
      • 07-29 08:18 in podman machine init machine init rootless docker.sock check
      • 07-29 08:18 in podman machine init machine init rootful with docker.sock check
      • 07-29 08:18 in podman machine ssh ssh to running machine and check os-type
      • 07-29 08:18 in podman machine rm Remove running machine
      • 07-29 08:18 in podman machine rm Remove machine sharing ssh key with another machine
      • 07-29 08:18 in podman machine reset reset with running machine and other machines idle
    • PR #23387
    • PR #23373
    • PR #23365
      • 07-22 16:18 in podman machine init machine init rootful with docker.sock check
    • PR #23360
    • PR #23275
      • 07-31 18:26 in 22551
      • 07-31 18:26 in 22551
      • 07-31 18:26 in podman machine start start simple machine
      • 07-31 18:26 in podman machine start start machine already started
      • 07-31 18:26 in podman machine start start machine with conflict on SSH port
      • 07-31 18:26 in podman machine start start only starts specified machine
      • 07-31 18:26 in podman machine start start two machines in parallel
      • 07-31 18:26 in podman machine init simple init with start
      • 07-31 18:26 in podman machine init machine init with cpus, disk size, memory, timezone
      • 07-31 18:26 in podman machine init machine init rootless docker.sock check
      • 07-31 18:26 in podman machine init machine init rootful with docker.sock check
      • 07-31 18:26 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 07-31 18:26 in podman machine set set machine cpus, disk, memory
      • 07-31 18:26 in podman machine set no settings should change if no flags
      • 07-31 18:26 in podman machine set set rootful with docker sock change
      • 07-31 18:26 in podman machine set set while machine already running
      • 07-29 17:01 in 22551
      • 07-29 17:01 in 22551
      • 07-29 17:01 in podman machine set set while machine already running
      • 07-29 17:01 in podman machine start start simple machine
      • 07-29 17:01 in podman machine start start machine already started
      • 07-29 17:01 in podman machine start start machine with conflict on SSH port
      • 07-29 17:01 in podman machine start start only starts specified machine
      • 07-29 17:01 in podman machine start start two machines in parallel
      • 07-29 17:01 in run basic podman commands Basic ops
      • 07-29 17:01 in run basic podman commands Volume should be disabled by command line
      • 07-29 17:01 in run basic podman commands Podman ops with port forwarding and gvproxy
      • 07-29 17:01 in run basic podman commands podman volume on non-standard path
      • 07-29 17:01 in podman machine init simple init with start
      • 07-29 17:01 in podman machine init machine init with cpus, disk size, memory, timezone
      • 07-29 17:01 in podman machine init machine init rootless docker.sock check
      • 07-29 17:01 in podman machine init machine init rootful with docker.sock check
      • 07-22 18:03 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 07-21 09:25 in 22551
      • 07-21 09:25 in 22551
      • 07-21 09:25 in podman machine set no settings should change if no flags
      • 07-21 09:25 in podman machine set set rootful with docker sock change
      • 07-21 09:25 in podman machine set set while machine already running
      • 07-18 14:26
      • 07-18 14:26
      • 07-18 14:26 in podman machine stop Stop running machine
      • 07-18 13:00 in podman machine init machine init with cpus, disk size, memory, timezone
  • windows : machine-wsl podman windows rootless host sqlite
    • PR #23418
      • 07-29 04:50 in podman machine rm Remove machine sharing ssh key with another machine
      • 07-29 04:50 in podman machine rm Remove machine sharing ssh key with another machine
      • 07-26 16:24 in podman inspect stop inspect two machines
      • 07-26 16:24 in podman inspect stop inspect with go format
    • PR #23398
      • 07-25 18:00 in podman machine rm Remove machine sharing ssh key with another machine
      • 07-25 18:00 in podman machine rm Removing all machines doesn't delete ssh keys
    • PR #23375
      • 07-23 10:54 in podman machine set set while machine already running
    • PR #23373
      • 07-23 14:53 in podman machine set set while machine already running
      • 07-23 14:53 in podman machine set set while machine already running
    • PR #23365
      • 07-22 16:04 in podman machine list list machine in human-readable format
    • PR #23360
      • 07-22 11:42 in podman inspect stop inspect with go format
      • 07-22 10:36 in podman machine reset reset with running machine and other machines idle
      • 07-22 10:36 in podman machine reset reset with running machine and other machines idle
      • 07-22 10:36 in podman machine reset reset with running machine and other machines idle
    • PR #23275
      • 07-31 18:36 in 22551
      • 07-31 18:36 in podman machine init - windows only init with user mode networking
      • 07-31 18:36 in 22551
      • 07-31 18:36 in 22551
      • 07-31 18:36 in 22551
      • 07-30 07:51 in podman machine start start two machines in parallel
      • 07-29 16:27 in podman machine start start two machines in parallel
      • 07-22 07:35 in podman machine init machine init rootless docker.sock check
      • 07-22 07:35 in podman machine init machine init rootful with docker.sock check
      • 07-19 08:05 in podman machine stop Stop running machine
      • 07-19 08:05 in podman machine stop Stop running machine
      • 07-19 08:05 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 07-19 08:05 in podman machine proxy settings propagation ssh to running machine and check proxy settings
      • 07-18 13:54 in podman machine set set while machine already running
x x x x x x
machine-hyperv(84) podman(164) windows(112) rootless(164) host(164) sqlite(164)
machine-mac(42) darwin(42)
machine-wsl(28) fedora-40-aarch64(10)
machine-linux(10)

edsantiago avatar Aug 01 '24 17:08 edsantiago