buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard

Panic during build

Open ciaranmcnulty opened this issue 3 years ago • 3 comments

Context: a docker compose build on a fresh checkout of a project. Some images were pulling, presumably something timed out when authenticating?

Hoping you can see what's happened from the line numbers maybe

 => => transferring context: 2B                                                                                                                                        0.0s
panic: send on closed channelrunner runtime_base  1/11] FROM public.ecr.aws/lambda/provided                                                                            0.0s
 => [[REDACTED]-mysql internal] load metadata for [REDACTED]/mysql:develop                                               1.7s
goroutine 466 [running]:tification-api-mock internal] load metadata for docker.io/wiremock/wiremock:2.31.0                                                             1.2s
github.com/docker/buildx/util/progress.(*Printer).Write(0x140001e7701?, 0x10142659c?)                                                                                  0.0s
 => [autgithub.com/docker/[email protected]/util/progress/printer.go:41 +0x2c                                                                                              0.0s
github.com/docker/buildx/build.Build.func3(0x1014d8ee3?)da-layer-cache:pull token for [REDACTED]                                                      0.0s
 => [autgithub.com/docker/[email protected]/build/build.go:681 +0x34 => [[REDACTED]-mysql internal] load .dockerignore                                                                                                          0.0s

github.com/moby/buildkit/util/progress/progresswriter.Wrap({0x140006e6770, 0x6d}, 0x1400000ca38, 0x140001e7920)
        github.com/moby/[email protected]/util/progress/progresswriter/progress.go: => => transferring context: 2B                                                                                                                                        0.0s
24 +0x1c8
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).Fetch => [[REDACTED]_lambda-runner internal] load metadata for public.ecr.aws/lambda/provided:latest                                                                  0.0s
(0x14000240990, {0x1019a35c8, 0x1400045eae0}, 0x140002bfbc0)
        github.com/moby/[email protected]/session/auth/authprovider/authprovider.go:88 +0x364
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1({0x1019a35c8, 0x1400045eae0}, {0x101951b40?, 0x140002bfbc0})
        github.com/moby/[email protected]/session/auth/auth.pb.go:954 +0x78
github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor({0x1019a35c8?, 0x1400045eae0?}, {0x101951b40?, 0x140002bfbc0?}, 0x14000746ad8?, 0x101170b08?)
        github.com/moby/[email protected]/util/grpcerrors/intercept.go:14 +0x38
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler({0x1018d66c0?, 0x14000240990}, {0x1019a35c8, 0x1400045eae0}, 0x140002bfb60, 0x10198f0e0)
        github.com/moby/[email protected]/session/auth/auth.pb.go:956 +0x13c
google.golang.org/grpc.(*Server).processUnaryRPC(0x140001d8000, {0x1019a88f8, 0x1400072fcb0}, 0x140000d9e60, 0x14000343620, 0x10218d718, 0x0)
        google.golang.org/[email protected]/server.go:1282 +0xb3c
google.golang.org/grpc.(*Server).handleStream(0x140001d8000, {0x1019a88f8, 0x1400072fcb0}, 0x140000d9e60, 0x0)
        google.golang.org/[email protected]/server.go:1619 +0x840
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/[email protected]/server.go:921 +0x88
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/[email protected]/server.go:919 +0x298
make: *** [test] Error 2

ciaranmcnulty avatar Sep 05 '22 15:09 ciaranmcnulty

Ah, I was not authenticated with the [REDACTED] registry - retrying gave me a 401 error instead

ciaranmcnulty avatar Sep 05 '22 15:09 ciaranmcnulty

So had a brief investigation: I think what's causing this is that the AuthProvider on the client is queried for some token, but at the point that we receive the request, we've now finished the build and shut down the progress writer - however the AuthProvider is still listening and handling incoming requests potentially.

Not quite sure, it's not immediately obvious why that channel has been closed.

jedevc avatar Sep 07 '22 13:09 jedevc

So this is actually a docker buildx issue but I can't transfer cross-org. The progress writer lib should make it sure that if the progresswriter is closed then running goroutines can't write to it. Also, all child goroutines should be closed before the progresswriter gets closed.

tonistiigi avatar Sep 09 '22 01:09 tonistiigi