buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard

Published 20 hours ago • verified publisher icon moby

fatal error: concurrent map writes in cache/remotecache/v1/chains.go

Open vlad-ivanov-name opened this issue 2 years ago • 14 comments

This is somewhat tricky to reproduce, but this happens during building two simultaneous builds, both using command line like this:

docker buildx build --tag image-tag --cache-to=type=inline \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:b6876ada103a6c19a663122c99ca61bdf189225cd4f010604b5708e5ffd97254 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:815a2b97b425e9aff078711c61958d1c582736c12020c5f31dd135bf70985cc8 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:23b0a217dbb1d0a68c8056c4b50a6e132982f06eeb8337671af06e31887d212e \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:86f77bb8c4e9156e3246344776a49242b714b6882efd9465ce2dc571dbd6e7e4 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:0c083c5937d7e90ab1f32dd85a49e6ce882464abf1537dc2cdef31649ea462cd \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:c9211caea0c6a54184b0238d73d2abddd5828c46f1cce1c4d116e12bae72e5bd \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:61b5cd93c9f1cc8fa75b406e150988b8e378a0da897b23d79f9e2411b48a4c73 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:f4c9f3069c0a5d7aa44dd053a67ba716a292a0dc994e30d1aeaedb5060015161 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:1121f2e59d30e14cfeea15830351e533edcc8965d0be56349d83b2c439ee1c5c \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:aa7f6a8f088b88b4b70d710390c38c2a27c40d956fe052b6730e63db17ec2bd3 \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:de316929b9b25987f4f951e61334ee1bf246d2843238d0ccc7795e75d857361b \
--cache-from=type=registry,ref=us.gcr.io/some-image@sha256:92fb7d0216f95ecc8fdd5ee26e0681d06f382ea06dc0c5508e25f4a76b4a78d4
.
Stack trace (truncated because too long for github): 
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: fatal error: concurrent map writes
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 263697 [running]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: runtime.throw({0x555746794ede, 0xc00141a5a8})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0045728a8 sp=0xc004572878 pc=0x555744fb48b1
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: runtime.mapassign_fast64ptr(0x555746d0a100, 0xc003970780, 0xc0009191a0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/runtime/map_fast64.go:266 +0x2cb fp=0xc0045728e0 sp=0xc0045728a8 pc=0x555744f8fa6b
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*item).LinkFrom(0xc0009191a0, {0x555747096920, 0xc003580a80}, 0x0, {0x0, 0x0})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:142 +0x185 fp=0xc004572950 sp=0xc0045728e0 pc=0x5557464bbc05
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000228b40, {0x5557470d6518, 0xc0028f3aa0}, {0x7f0620683f60, 0xc000d58228}, {0xc00638a4b0, 0xeda17e381, {0x555747087820, 0xc00638a468}})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:159 +0xa3d fp=0xc004572c30 sp=0xc004572950 pc=0x5557460c7f7d
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0005a7bd0, {0x5557470d6518, 0xc0028f3aa0}, {0x7f0620683f60, 0xc000d58228}, {0xc00638a4b0, 0xeda17e382, {0x555747087820, 0xc00638a468}})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc004572f10 sp=0xc004572c30 pc=0x5557460c836f
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000229db0, {0x5557470d6518, 0xc0071b77a0}, {0x7f0620683f60, 0xc000d58228}, {0xc00638a4b0, 0xc00638a468, {0x555747087820, 0xc00638a468}})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc0045731f0 sp=0xc004572f10 pc=0x5557460c836f
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache({0x5557470d6518, 0xc0071b77a0}, {0x5557470d71c8, 0xc000d58228}, {0x5557470dceb8, 0xc004a69680}, {0x555747087820, 0xc00638a468})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:287 +0x2c6 fp=0xc004573318 sp=0xc0045731f0 pc=0x5557464ff426
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc0013d8300, {0x5557470d6518, 0xc0071b77a0}, {0xc0061ec0a0, 0x19}, {0xc0061ec0c0, 0x5}, {0x0, 0x0, {0x0, ...}, ...}, ...)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0xbc6 fp=0xc004573888 sp=0xc004573318 pc=0x5557464fd966
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc00140d200, {0x5557470d6518, 0xc0071b77a0}, 0xc002a0c5a0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x9e5 fp=0xc004573b50 sp=0xc004573888 pc=0x555746505925
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Solve_Handler({0x555746ef3c20, 0xc00140d200}, {0x5557470d6518, 0xc0071b77a0}, 0xc007144660, 0x0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1482 +0x170 fp=0xc004573ba8 sp=0xc004573b50 pc=0x5557460888b0
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000494600, {0x5557470feaf8, 0xc00183f830}, 0xc0067a1f00, 0xc001185b30, 0x5557482c8818, 0x0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1024 +0xd1e fp=0xc004573e48 sp=0xc004573ba8 pc=0x5557456c4a9e
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000494600, {0x5557470feaf8, 0xc00183f830}, 0xc0067a1f00, 0x0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1313 +0xa56 fp=0xc004573f68 sp=0xc004573e48 pc=0x5557456c82d6
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:722 +0x98 fp=0xc004573fe0 sp=0xc004573f68 pc=0x5557456c2c58
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: runtime.goexit()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc004573fe8 sp=0xc004573fe0 pc=0x555744fe8e21
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:720 +0xef
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 1 [chan receive, 25 minutes]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: main.(*DaemonCli).start(0xc000cb5b60, 0xc0000b27e0)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0xf2b
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: main.runDaemon(...)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: main.newDaemonCommand.func1(0xc000992000, {0xc000cb5b00, 0x3, 0x3})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x5c
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc000992000, {0xc000132010, 0x3, 0x3})
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x60e
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000992000)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: main.main()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x15d
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 10 [select]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0000bc410)
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:154 +0xb9
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:32 +0x87
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 16 [syscall]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: os/signal.signal_recv()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/runtime/sigqueue.go:169 +0x98
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: os/signal.loop()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/os/signal/signal_unix.go:24 +0x19
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: created by os/signal.Notify.func1.1
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /usr/local/go/src/os/signal/signal.go:151 +0x2c
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 98 [chan receive]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/pkg/signal.Trap.func1()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x65
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: created by github.com/docker/docker/pkg/signal.Trap
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x165
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: goroutine 86 [chan receive, 25 minutes]:
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]: github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1()
May 31 08:52:31 terraform-vm-jenkins-agent dockerd[27041]:         /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:19 +0x58

Docker version:

docker --version
Docker version 20.10.16, build aa7e414
containerd --version
containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16

vlad-ivanov-name avatar May 31 '22 07:05 vlad-ivanov-name

What version of buildkit is this(meaning what driver do you use in buildx)? If it is embedded buildkit in 20.10.16 then that is v0.8 .

tonistiigi avatar May 31 '22 15:05 tonistiigi

It is the embedded one, not sure how to check version but it's the default builder with docker driver

docker buildx inspect default
Name:   default
Driver: docker

Nodes:
Name:      default
Endpoint:  default
Status:    running
Platforms: linux/amd64, linux/386

vlad-ivanov-name avatar May 31 '22 15:05 vlad-ivanov-name

I saw that buildkit had a commit: https://github.com/moby/buildkit/commit/ba90a9ee337b8d53dba25ea82a8b17c4090540e8 to add a mutex around backlinks, but it's not vendored in 20.10: https://github.com/moby/moby/blob/20.10/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go

I'm trying to build 20.10 with an additional mutex for links and backported patch for backlinks

vlad-ivanov-name avatar May 31 '22 15:05 vlad-ivanov-name

Backporting the patch from https://github.com/moby/buildkit/commit/ba90a9ee337b8d53dba25ea82a8b17c4090540e8 changes where the issue occurs:

Stack trace 
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: fatal error: concurrent map writes
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: goroutine 141500 [running]:
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: runtime.throw({0x55b03214aede, 0xc0008dbba8})
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc002f3ed88 sp=0xc002f3ed58 pc=0x55b03096a8b1
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: runtime.mapassign(0xc000bcef50, 0xc002f446a8, 0xc001890320)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /usr/local/go/src/runtime/map.go:585 +0x4d6 fp=0xc002f3ee08 sp=0xc002f3ed88 pc=0x55b030942c56
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000bcfdc0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:207 +0x730 fp=0xc002f3f110 sp=0xc002f3ee08 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000c5c0e0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f3f418 sp=0xc002f3f110 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000c5ca10, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f3f720 sp=0xc002f3f418 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc00072f6c0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f3fa28 sp=0xc002f3f720 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc00072fdc0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f3fd30 sp=0xc002f3fa28 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a40e0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40038 sp=0xc002f3fd30 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a48c0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40340 sp=0xc002f40038 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a4f50, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40648 sp=0xc002f40340 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444c40, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40950 sp=0xc002f40648 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a5420, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40c58 sp=0xc002f40950 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444b60, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f40f60 sp=0xc002f40c58 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a5960, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f41268 sp=0xc002f40f60 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0002a5b20, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f41570 sp=0xc002f41268 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444a10, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f41878 sp=0xc002f41570 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0003f61c0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f41b80 sp=0xc002f41878 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0003f63f0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f41e88 sp=0xc002f41b80 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0004448c0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f42190 sp=0xc002f41e88 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0003f6b60, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f42498 sp=0xc002f42190 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0003f6e70, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f427a0 sp=0xc002f42498 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0003f70a0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f42aa8 sp=0xc002f427a0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444620, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f42db0 sp=0xc002f42aa8 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0004445b0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f430b8 sp=0xc002f42db0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444540, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f433c0 sp=0xc002f430b8 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0004444d0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f436c8 sp=0xc002f433c0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc00036a9a0, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f439d0 sp=0xc002f436c8 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444380, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f43cd8 sp=0xc002f439d0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc00036af50, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f43fe0 sp=0xc002f43cd8 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc00036b260, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f442e8 sp=0xc002f43fe0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc000444230, 0xc002f446a8)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc002f445f0 sp=0xc002f442e8 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*CacheChains).normalize(0xc0016c8280)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:62 +0x3ce fp=0xc002f449b8 sp=0xc002f445f0 pc=0x
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*CacheChains).Marshal(0xc0016c8280)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:77 +0x33 fp=0xc002f44a78 sp=0xc002f449b8 pc=0x5
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/inline.(*exporter).ExportForLayers(0xc000ea81c8, {0xc00080d680, 0x26, 0x7ffa
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/inline/inline.go:42 +0x4d fp=0xc002f451f0 sp=0xc002f44a78 pc
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache({0x55b032a8c4b8, 0xc00193e9c0}, {0x55b032a8d168, 0xc000ea81c8}, {
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:295 +0x307 fp=0xc002f45318 sp=0xc002f451f0 pc=0x55b
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc001036a80, {0x55b032a8c4b8, 0xc00193e9c0}, {0xc00230e180,
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0xbc6 fp=0xc002f45888 sp=0xc002f45318 pc=0x55b
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc000fc6240, {0x55b032a8c4b8, 0xc00193e9c0}, 0xc002cfe120)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x9e5 fp=0xc002f45b50 sp=0xc002f45888 pc=0x55b031ebb9e
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Solve_Handler({0x55b0328a9bc0, 0xc000fc6240}, {0x55b032a8c4b8, 0
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1482 +0x170 fp=0xc002f45ba8 sp=0xc002f45b50
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004b6180, {0x55b032ab4a98, 0xc00180c990}, 0xc0000d8900, 0xc0010428
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1024 +0xd1e fp=0xc002f45e48 sp=0xc002f45ba8 pc=0x55b03107aa9e
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0004b6180, {0x55b032ab4a98, 0xc00180c990}, 0xc0000d8900, 0x0)
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1313 +0xa56 fp=0xc002f45f68 sp=0xc002f45e48 pc=0x55b03107e2d6
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:722 +0x98 fp=0xc002f45fe0 sp=0xc002f45f68 pc=0x55b031078c58
Jun 01 10:47:41 terraform-vm-jenkins-agent dockerd[851]: runtime.goexit()

This is the line, adjusted according to the patch applied:

https://github.com/moby/moby/blob/90033ab093a3b50719f59b8a554527f69f7442b2/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go#L59

vlad-ivanov-name avatar Jun 01 '22 08:06 vlad-ivanov-name

I tried putting mutexes around some of the objects accessed concurrently, from smaller lock regions to bigger (item.links, CacheChains.items), but somehow none of my attempts worked and I was still getting crashes originating from normalizeItem. I ended up wrapping the whole inline cache export in mutex, which I realise is probably an overkill, but unfortunately I'm not familiar enough with the code. My best guess is that a proper patch would create a lock for each map individually, or maybe use a sync.Map. Here's a patch that fixed the issue for me:

diff --git a/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go b/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go
index 31e8afca7a..74de8cc5f0 100644
--- a/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go
+++ b/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go
@@ -5,6 +5,7 @@ import (
        "fmt"
        "strings"
        "time"
+       "sync"

        "github.com/moby/buildkit/cache"
        "github.com/moby/buildkit/cache/remotecache"
@@ -46,6 +47,7 @@ type Solver struct {
        gatewayForwarder          *controlgateway.GatewayForwarder
        sm                        *session.Manager
        entitlements              []string
+       cacheMu                   sync.Mutex
 }

 func New(wc *worker.Controller, f map[string]frontend.Frontend, cache solver.CacheManager, resolveCI map[string]remotecache.ResolveCacheImporterFunc, gatewayForwarder *controlgateway.GatewayForwarder, sm *session.Manager, ents []string) (*Solver, error) {
@@ -173,7 +175,9 @@ func (s *Solver) Solve(ctx context.Context, id string, sessionID string, req fro
                        }
                        inp.Ref = workerRef.ImmutableRef

+                       s.cacheMu.Lock()
                        dt, err := inlineCache(ctx, exp.CacheExporter, r, session.NewGroup(sessionID))
+                       s.cacheMu.Unlock()
                        if err != nil {
                                return nil, err
                        }
@@ -197,7 +201,9 @@ func (s *Solver) Solve(ctx context.Context, id string, sessionID string, req fro
                                        }
                                        m[k] = workerRef.ImmutableRef

+                                       s.cacheMu.Lock()
                                        dt, err := inlineCache(ctx, exp.CacheExporter, r, session.NewGroup(sessionID))
+                                       s.cacheMu.Unlock()
                                        if err != nil {
                                                return nil, err
                                        }

vlad-ivanov-name avatar Jun 01 '22 14:06 vlad-ivanov-name

I wonder how slow can inline cache export process be? If it's just walking a tree without moving layers around maybe it's okay to lock it like that.

vlad-ivanov-name avatar Jun 01 '22 14:06 vlad-ivanov-name

I am facing a similar problem when running multiple (around 20) concurrent builds in GitLab CI. I also use --cache-from (without it it works fine).

> docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:17 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

> docker buildx version
github.com/docker/buildx v0.8.2-docker 6224def4dd2c3d347eee19db595348c50d7cb491

My command looks like this:

docker build \
      --cache-from $DOCKER_IMAGE:$BRANCH_TAG \
      --cache-from $DOCKER_IMAGE:$CI_COMMIT_SHORT_SHA \
      -t $DOCKER_IMAGE:$BRANCH_TAG \
      -t $DOCKER_IMAGE:$CI_COMMIT_SHORT_SHA \
      -t $DOCKER_IMAGE:$TIMESTAMP_TAG \
      --build-arg APP=$APP \
      --build-arg BUILDKIT_INLINE_CACHE=1 \
      --secret id=settings,src=settings-secret.xml \
      .

Dockerfile has two stages:

# syntax=docker/dockerfile:1.2

FROM maven:3.8.5-openjdk-17-slim as build-stage
ARG APP
WORKDIR /$APP
COPY $APP/pom.xml .
COPY $APP/src src
RUN --mount=type=secret,id=settings,dst=/root/.m2/settings.xml \
    mvn package -U -Dmaven.test.skip

FROM openjdk:17-alpine
ARG APP
WORKDIR /$APP
COPY --from=build-stage \
    /$APP/target/$APP.jar $APP.jar
CMD ...

Truncated stack trace from daemon.log (I can provide a full version too):

Jun 20 11:06:46 [server_name] dockerd[2231560]: fatal error: concurrent map writes
Jun 20 11:06:46 [server_name] dockerd[2231560]: goroutine 324499 [running]:
Jun 20 11:06:46 [server_name] dockerd[2231560]: runtime.throw({0x561db4a3eeee, 0x0})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc00237c468 sp=0xc00237c438 pc=0x561db325e8b1
Jun 20 11:06:46 [server_name] dockerd[2231560]: runtime.mapassign_fast64ptr(0x561db4fb85e0, 0xc005cbb080, 0xc005eeff80)
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /usr/local/go/src/runtime/map_fast64.go:191 +0x2dc fp=0xc00237c4a0 sp=0xc00237c468 pc=0x561db3239a7c
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*item).LinkFrom(0xc005eeff80, {0x561db53416a0, 0xc004324000}, 0x0, {0x0, 0x0})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:142 +0x185 fp=0xc00237c510 sp=0xc00237c4a0 pc=0x561db4765fa5
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.addBacklinks.func1({0xc0043c4550, 0x47}, {0x0, 0x0, {0xc0043c48c0, 0x47}, {0x0, 0x0}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:42 +0x169 fp=0xc00237c588 sp=0xc00237c510 pc=0x561db4371849
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*cacheKeyStorage).WalkBacklinks(0x561db4fc40a0, {0xc0057bc381, 0x19}, 0xc005cbb0b0)
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/cachestorage.go:165 +0x269 fp=0xc00237c718 sp=0xc00237c588 pc=0x561db47641a9
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.addBacklinks({0x7fca902dc900, 0xc000eac7f8}, {0x561db53416a0, 0xc005eeff80}, 0xc01041b720, {0xc0057bc381, 0x19}, 0xc003ae3260)
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:29 +0x1d7 fp=0xc00237c750 sp=0xc00237c718 pc=0x561db43715b7
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc007f7e0f0, {0x561db53812d8, 0xc003ae3290}, {0x7fca902dc900, 0xc000eac7f8}, {0xc00b4d0870, 0x0, {0x561db53325a0, 0xc00b4d0858}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:164 +0x7e7 fp=0xc00237ca30 sp=0xc00237c750 pc=0x561db43720c7
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000b6e2d0, {0x561db53812d8, 0xc003ae3290}, {0x7fca902dc900, 0xc000eac7f8}, {0xc00b4d0870, 0x0, {0x561db53325a0, 0xc00b4d0858}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc00237cd10 sp=0xc00237ca30 pc=0x561db437270f
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00d77c460, {0x561db53812d8, 0xc003ae3290}, {0x7fca902dc900, 0xc000eac7f8}, {0xc00b4d0870, 0x19, {0x561db53325a0, 0xc00b4d0858}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc00237cff0 sp=0xc00237cd10 pc=0x561db437270f
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000a6a780, {0x561db53812d8, 0xc003ae3290}, {0x7fca902dc900, 0xc000eac7f8}, {0xc00b4d0870, 0xeda422ea8, {0x561db53325a0, 0xc00b4d0858}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc00237d2d0 sp=0xc00237cff0 pc=0x561db437270f
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc000b6f3b0, {0x561db5381230, 0xc00f8110c0}, {0x7fca902dc900, 0xc000eac7f8}, {0xc00b4d0870, 0xc00b4d0858, {0x561db53325a0, 0xc00b4d0858}})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xe2f fp=0xc00237d5b0 sp=0xc00237d2d0 pc=0x561db437270f
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache({0x561db5381230, 0xc00f8110c0}, {0x561db5381f88, 0xc000eac7f8}, {0x561db5387c78, 0xc001a732a0}, {0x561db53325a0, 0xc00b4d0858})
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:287 +0x2c6 fp=0xc00237d6d8 sp=0xc00237d5b0 pc=0x561db47a97c6
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc0011a7980, {0x561db5381230, 0xc00f8110c0}, {0xc00a2905c1, 0x19}, {0xc003ba8270, 0x60}, {0x0, 0x0, {0x561db4a2bfb2, ...}, ...}, ...)
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0xbc6 fp=0xc00237dc48 sp=0xc00237d6d8 pc=0x561db47a7d06
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc000924f30, {0x561db5381230, 0xc00f8110c0}, 0xc001a96c60)
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x9e5 fp=0xc00237df10 sp=0xc00237dc48 pc=0x561db47afcc5
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/builder/builder-next.(*Builder).Build.func2()
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/builder/builder-next/builder.go:376 +0x54 fp=0xc00237df78 sp=0xc00237df10 pc=0x561db4837f74
Jun 20 11:06:46 [server_name] dockerd[2231560]: github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
Jun 20 11:06:46 [server_name] dockerd[2231560]:  /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x67 fp=0xc00237dfe0 sp=0xc00237df78 pc=0x561db34ea147
[...]

maciek16180 avatar Jun 20 '22 13:06 maciek16180

Hi all, same problem here.

I'm using gitlab-runner with docker executor to build some docker images with the following docker-compose -f docker-compose-build.yml build command.

Randomly, docker daemon crashes and restarts.

logs 
Jul 06 08:29:22 appdev01 dockerd[178101]: fatal error: concurrent map writes
Jul 06 08:29:22 appdev01 dockerd[178101]: goroutine 9693 [running]:
Jul 06 08:29:22 appdev01 dockerd[178101]: runtime.throw({0x56417d660eed, 0xc0012bd2b8})
Jul 06 08:29:22 appdev01 dockerd[178101]:         /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc001259658 sp=0xc001259628 pc=0x56417be808d1
Jul 06 08:29:22 appdev01 dockerd[178101]: runtime.mapassign(0xc0008f3d40, 0xc00125a6a8, 0xc002435920)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /usr/local/go/src/runtime/map.go:585 +0x4d6 fp=0xc0012596d8 sp=0xc001259658 pc=0x56417be58c76
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc002435920, 0xc00125a6a8)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:207 +0x730 fp=0xc0012599e0 sp=0xc0012596d8 pc=0x56417d38ab50
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0008f3c80, 0xc00125a6a8)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc001259ce8 sp=0xc0012599e0 pc=0x56417d38aea5
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0008f3c20, 0xc00125a6a8)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc001259ff0 sp=0xc001259ce8 pc=0x56417d38aea5
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0008f3bc0, 0xc00125a6a8)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc00125a2f8 sp=0xc001259ff0 pc=0x56417d38aea5
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.normalizeItem(0xc0008f3b60, 0xc00125a6a8)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:158 +0xa85 fp=0xc00125a600 sp=0xc00125a2f8 pc=0x56417d38aea5
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*CacheChains).normalize(0xc002a8e520)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:59 +0x367 fp=0xc00125a9b8 sp=0xc00125a600 pc=0x56417d3879e7
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*CacheChains).Marshal(0xc002a8e520)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:74 +0x33 fp=0xc00125aa78 sp=0xc00125a9b8 pc=0x56417d387bb3
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/inline.(*exporter).ExportForLayers(0xc000c16528, {0xc0019a6000, 0x1a, 0x7f1d3c535b80})
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/inline/inline.go:42 +0x4d fp=0xc00125b1f0 sp=0xc00125aa78 pc=0x56417d41518d
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache({0x56417dfa32d8, 0xc0011f9830}, {0x56417dfa3f88, 0xc000c16528}, {0x56417dfa9c78, 0xc002f274e0}, {0x56417df545a
0, 0xc001fe2408})
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:295 +0x307 fp=0xc00125b318 sp=0xc00125b1f0 pc=0x56417d3cb827
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc000659440, {0x56417dfa32d8, 0xc0011f9830}, {0xc0027904c0, 0x19}, {0xc0027904e0, 0xc0011f9800}, {0x0, 0x
0, {0x0, ...}, ...}, ...)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0xbc6 fp=0xc00125b888 sp=0xc00125b318 pc=0x56417d3c9d26
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc000b3d710, {0x56417dfa32d8, 0xc0011f9830}, 0xc001cd0120)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x9e5 fp=0xc00125bb50 sp=0xc00125b888 pc=0x56417d3d1ce5
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Solve_Handler({0x56417ddc09a0, 0xc000b3d710}, {0x56417dfa32d8, 0xc0011f9830}, 0xc001492780, 0x0)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1482 +0x170 fp=0xc00125bba8 sp=0xc00125bb50 pc=0x56417cf54c70
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ce000, {0x56417dfcb8b8, 0xc00278db90}, 0xc0012eb700, 0xc0006652f0, 0x56417f195818, 0x0)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1024 +0xd1e fp=0xc00125be48 sp=0xc00125bba8 pc=0x56417c590c9e
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0001ce000, {0x56417dfcb8b8, 0xc00278db90}, 0xc0012eb700, 0x0)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1313 +0xa56 fp=0xc00125bf68 sp=0xc00125be48 pc=0x56417c5944d6
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:722 +0x98 fp=0xc00125bfe0 sp=0xc00125bf68 pc=0x56417c58ee58
Jul 06 08:29:22 appdev01 dockerd[178101]: runtime.goexit()
Jul 06 08:29:22 appdev01 dockerd[178101]:         /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00125bfe8 sp=0xc00125bfe0 pc=0x56417beb4e41
Jul 06 08:29:22 appdev01 dockerd[178101]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:720 +0xef
Jul 06 08:29:22 appdev01 dockerd[178101]: goroutine 1 [chan receive, 18 minutes]:
Jul 06 08:29:22 appdev01 dockerd[178101]: main.(*DaemonCli).start(0xc000726810, 0xc0002065a0)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0xf2b
Jul 06 08:29:22 appdev01 dockerd[178101]: main.runDaemon(...)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
Jul 06 08:29:22 appdev01 dockerd[178101]: main.newDaemonCommand.func1(0xc0001db340, {0xc0007267b0, 0x3, 0x3})
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x5c
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc0001db340, {0xc00004e050, 0x3, 0x3})
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x60e
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0001db340)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
Jul 06 08:29:22 appdev01 dockerd[178101]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
Jul 06 08:29:22 appdev01 dockerd[178101]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895

----- truncated -----

Jul 05 16:25:29 appdev01 dockerd[23023]: goroutine 19411 [semacquire]:
Jul 05 16:25:29 appdev01 dockerd[23023]: sync.runtime_Semacquire(0xc001425d40)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /usr/local/go/src/runtime/sema.go:56 +0x25
Jul 05 16:25:29 appdev01 dockerd[23023]: sync.(*WaitGroup).Wait(0xc002607500)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /usr/local/go/src/sync/waitgroup.go:130 +0x71
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc000fb20f0)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x27
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000b73b90, 0xc001161ce0, {0x563e1f583840, 0xc0011b62b0})
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1b3
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control._Control_Status_Handler({0x563e1f37e9a0, 0xc000b73b90}, {0x563e1f569e68, 0xc0014e8000})
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/api/services/control/control.pb.go:1499 +0xd0
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00075a300, {0x563e1f5898b8, 0xc0020035f0}, 0xc0008bd800, 0xc00072cff0, 0x563e20757720, 0x0)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1237 +0xd63
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00075a300, {0x563e1f5898b8, 0xc0020035f0}, 0xc0008bd800, 0x0)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:1317 +0xa10
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:722 +0x98
Jul 05 16:25:29 appdev01 dockerd[23023]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/server.go:720 +0xef
Jul 05 16:25:29 appdev01 dockerd[23023]: goroutine 19407 [chan receive]:
Jul 05 16:25:29 appdev01 dockerd[23023]: github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000b70b00)
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x48d
Jul 05 16:25:29 appdev01 dockerd[23023]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
Jul 05 16:25:29 appdev01 dockerd[23023]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x147
Jul 05 16:25:29 appdev01 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
`docker info` 
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-121-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.43GiB
 Name: xxxxx
 ID: JQ62:PDO5:VUOF:ESI5:VNTU:NVUN:4UBZ:J3AT:VT64:NHEW:MX2I:7CL3
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 26
  Goroutines: 34
  System Time: 2022-07-06T10:02:00.795104833Z
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://xxxxx/
 Live Restore Enabled: false

WARNING: No swap limit support
`docker-compose-build.yml` 
version: '3.8'

x-build-common:
  &build-common
  context: .
  dockerfile: Dockerfile
  cache_from:
    - ${IMG_DEV_LATEST}
    - ${IMG_DEVELOP_DEV_LATEST}
  args:
    - BUILDKIT_INLINE_CACHE=1

services:
  dev_latest:
    build:
      <<: *build-common
      target: dev
    image: ${IMG_DEV_LATEST}
  dev_sha:
    build:
      <<: *build-common
      target: dev
    image: ${IMG_DEV_SHA}
  hub_latest:
    build:
      <<: *build-common
      target: hub
    image: ${IMG_HUB_LATEST}
  hub_sha:
    build:
      <<: *build-common
      target: hub
    image: ${IMG_HUB_SHA}

vb-quaestio avatar Jul 06 '22 10:07 vb-quaestio

Any update on this issue? This is quite frustrating, we just don't use --cache-from in the pipeline because of that :(

maciek16180 avatar Jul 27 '22 08:07 maciek16180

We've been running docker with the patch from https://github.com/moby/buildkit/issues/2889#issuecomment-1143715782 for around two months now, a few thousand images were built, and I haven't noticed any significant issues. There were no crashes related to cache.

vlad-ivanov-name avatar Jul 27 '22 08:07 vlad-ivanov-name

Any update on this issue? This is quite frustrating, we just don't use --cache-from in the pipeline because of that :(

A work around that worked for me was to use single target builds. So, instead of running docker-compose -f docker-compose-build.yml build, i'm running

docker-compose -f docker-compose-build.yml build target1
docker-compose -f docker-compose-build.yml build target2
docker-compose -f docker-compose-build.yml build targetx

vb-quaestio avatar Jul 27 '22 09:07 vb-quaestio

Unfortunately, both of those workarounds can't be applied in our case. And even then, they are workarounds. I would hope for a proper fix of the problem.

maciek16180 avatar Jul 27 '22 20:07 maciek16180

If there is a reproducer, then please try it against Buildkit v0.10 (or Docker 22.06-beta). It's likely this has been already fixed.

tonistiigi avatar Jul 27 '22 22:07 tonistiigi

Oh, cool. It's hard for me to change Docker version just for testing though. I think I will wait until 22 is released, we will update and see it is works then. Thanks.

maciek16180 avatar Jul 28 '22 12:07 maciek16180

We've been running docker with the patch from [#2889 (comment)].

@vlad-ivanov-name Seems that patching our Docker would be the best available solution so far (as no release date for 22.0 yet), do you have some insights to do so?

Did you build https://github.com/moby/moby/tree/20.10 branch & replaced number of binaries on target machines? Any other step you went through?

AhmedMozaly avatar Dec 01 '22 14:12 AhmedMozaly

@AhmedMozaly I build packages using code in https://github.com/docker/docker-ce-packaging/tree/20.10 . It needs to be patched to remove checkout dependency in Makefile so that you can checkout, patch, and then build

diff --git a/Makefile b/Makefile
index 80cdae1386..f1a6f1dd0b 100644
--- a/Makefile
+++ b/Makefile
@@ -80,20 +80,20 @@ clean: clean-src ## remove build artifacts
 	$(MAKE) -C static clean
 
 .PHONY: deb rpm
-deb rpm: checkout ## build rpm/deb packages
+deb rpm: ## build rpm/deb packages
 	$(MAKE) -C $@ VERSION=$(VERSION) GO_VERSION=$(GO_VERSION) $@
 
 .PHONY: centos-% fedora-% rhel-%
-centos-% fedora-% rhel-%: checkout ## build rpm packages for the specified distro
+centos-% fedora-% rhel-%: ## build rpm packages for the specified distro
 	$(MAKE) -C rpm VERSION=$(VERSION) GO_VERSION=$(GO_VERSION) $@
 
 .PHONY: debian-% raspbian-% ubuntu-%
-debian-% raspbian-% ubuntu-%: checkout ## build deb packages for the specified distro
+debian-% raspbian-% ubuntu-%: ## build deb packages for the specified distro
 	$(MAKE) -C deb VERSION=$(VERSION) GO_VERSION=$(GO_VERSION) $@
 
 .PHONY: static
 static: DOCKER_BUILD_PKGS:=static-linux cross-mac cross-win cross-arm
-static: checkout ## build static-compiled packages
+static: ## build static-compiled packages
 	for p in $(DOCKER_BUILD_PKGS); do \
 		$(MAKE) -C $@ VERSION=$(VERSION) GO_VERSION=$(GO_VERSION) TARGETPLATFORM=$(TARGETPLATFORM) CONTAINERD_VERSION=$(CONTAINERD_VERSION) RUNC_VERSION=$(RUNC_VERSION) $${p}; \
 	done

vlad-ivanov-name avatar Dec 01 '22 14:12 vlad-ivanov-name