tilt icon indicating copy to clipboard operation
tilt copied to clipboard

tilt startup hangs on dead docker connection

Open nicks opened this issue 3 years ago • 1 comments

Repro steps:

  • Create a socket file with a dead listener by running:
nc -lkU /home/nick/test.sock
  • Start tilt and try to connect to the socket:
DOCKER_HOST="unix:///home/nick/test.sock" tilt ci

Expected behavior: Tilt should notice that it's not talking to a well-formed Docker daemon and error out and/or time out.

Actual behavior: Tilt hangs indefinitely on startup.

Received this stack trace from a user that pretty clearly points to the problem:

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0xc000c530e0, 0xc0005a5a80)
	/usr/local/go/src/net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0xc00090cf00, 0xc000516b00)
	/usr/local/go/src/net/http/transport.go:594 +0x7c9
net/http.(*Transport).RoundTrip(0x400e285?, 0x7258180?)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc000516b00, {0x7258180, 0xc00090cf00}, {0x68a5180?, 0x1?, 0x0?})
	/usr/local/go/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc000a89ad0, 0xc000516b00, {0x90?, 0x322fffff?, 0x0?})
	/usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000a89ad0, 0xc000516b00)
	/usr/local/go/src/net/http/client.go:725 +0x8f5
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:593
github.com/docker/docker/client.(*Client).doRequest(0xc0003f5d80, {0x7281780, 0xc000054098}, 0xc000516900)
	/root/project/vendor/github.com/docker/docker/client/request.go:125 +0x288
github.com/docker/docker/client.(*Client).Ping(0xc0003f5d80, {0x7281780, 0xc000054098})
	/root/project/vendor/github.com/docker/docker/client/ping.go:26 +0x136
github.com/docker/cli/cli/command.(*DockerCli).initializeFromClient(0xc000c15040)
	/root/project/vendor/github.com/docker/cli/cli/command/cli.go:361 +0x11d
github.com/docker/cli/cli/command.(*DockerCli).Initialize(0xc000c15040, 0xc000c50de0, {0x0, 0x0, 0xc000cb2278?})
	/root/project/vendor/github.com/docker/cli/cli/command/cli.go:267 +0x626
github.com/tilt-dev/tilt/internal/docker.RealClientCreator.FromCLI({}, {0x69405d1?, 0xb?})
	/root/project/internal/docker/env.go:122 +0x1ed
github.com/tilt-dev/tilt/internal/docker.ProvideClusterEnv({0x72817f0, 0xc0008bef90}, {0x72688e8, 0x8a4afb8}, {0xc00037c400, 0x3d}, {0x692ed3e, 0x3}, {0x6933eb7, 0x6}, ...)
	/root/project/internal/docker/env.go:250 +0x82a
github.com/tilt-dev/tilt/internal/cli.wireCmdUp({0x72817f0?, 0xc0008bef90}, 0xc0009083c0, 0xc0008bef30, {0x692e4c2, 0x2})
	/root/project/internal/cli/wire_gen.go:288 +0x1c57
github.com/tilt-dev/tilt/internal/cli.(*upCmd).run(0xc00089a090, {0x7281748?, 0xc000908a40?}, {0xc0008bd640, 0x0, 0x2})
	/root/project/internal/cli/up.go:148 +0x5ce
github.com/tilt-dev/tilt/internal/cli.addCommand.func1(0xc000a49900?, {0xc0008bd640, 0x0, 0x2})
	/root/project/internal/cli/cli.go:156 +0xb1
github.com/spf13/cobra.(*Command).execute(0xc000a49900, {0xc0008bd620, 0x2, 0x2})
	/root/project/vendor/github.com/spf13/cobra/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xc000a49400)
	/root/project/vendor/github.com/spf13/cobra/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/root/project/vendor/github.com/spf13/cobra/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/root/project/vendor/github.com/spf13/cobra/command.go:895
github.com/tilt-dev/tilt/internal/cli.Execute()
	/root/project/internal/cli/cli.go:96 +0xece
main.main()
	/root/project/cmd/tilt/main.go:19 +0xde

nicks avatar Jun 03 '22 05:06 nicks

actually, i can reproduce this with Docker CLI too, i think the bug is here:

https://github.com/docker/cli/blob/master/cli/command/cli.go#L316

where it doesn't attach any timeout to the initial unix socket connection

nicks avatar Jun 03 '22 05:06 nicks