Alex Couture-Beil

Here's some interesting printf logs related a cancel: ``` b.builder.Build def=0xc00029eb40 digest=sha256:0cba2c4356ace9612563191c8f6f3c21dad9777550b693dfdb0d47dcd53e31b1 returned err=context canceled panic: got an error when calling build on def=0xc00029eb40 digest=sha256:0cba2c4356ace9612563191c8f6f3c21dad9777550b693dfdb0d47dcd53e31b1 ``` First time adding a...

I tried adding reference counting between the job / state link under https://github.com/earthly/buildkit/commit/c1a059419de096a23d121451ffcd7c08804ab260 but that didn't help.

I tried adding a [lock to the job struct](https://github.com/earthly/buildkit/commit/8b09f46a632a1b0661cb73654c85c30b52dd7a0e) and that also didn't help.

I had a look at merging issues, to see if there was an obvious connection, but doesn't appear to be. ``` 931 first context cancel 2001 first time 0cba2c4356ace9612563191c8f6f3c21dad9777550b693dfdb0d47dcd53e31b1 is...

the reproduction case previously shared was invalid; ad6badc5af52f7a8b1cacb2e867731004b77581e contains a valid reproduction under the +breakit target, which doesn't always get hit, therefore one must put it in a retry wrapper:...

I noticed that closeCh is sometimes closed before the builder.Build function completes; I will take a closer look to see if there's a race here e.g. this log: ``` time="2024-02-28T00:41:30Z"...

some client-side logs related to the session being closed: ``` ACB creating buildFunc Build 🔧 ———————————————————————————————————————————————————————————————————————————————— ACB calling bkClient.Build NewSession called session.Run called by goroutine 188 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e...

however when I look on the buildkit logs that go along with the above client logs, I see this ordering: ``` time="2024-02-28T20:17:05Z" level=debug msg="getImmutableRef called on qakktds5cwk1nagzdabktaxxx / context is...

``` forceExecution /home/alex/junk/earthly-fake-tests+medium calling reference.ReadDir kfe1bl9k4nxihikfc0c10l6qf / lLBBridgeClient.ReadDir Ref:"kfe1bl9k4nxihikfc0c10l6qf" DirPath:"/" RunExitCode /home/alex/junk/earthly-fake-tests+breakit lLBBridgeClient.ReadFile Ref:"pclwwgbajdhek4wq3bm8inm9c" FilePath:"/tmp/earthly_if_statement_exit_code" /h/a/j/earthly-fake-tests+medium | DOCKERHUB_MIRROR_AUTH_FROM_CLOUD_SECRETS=true /h/a/j/earthly-fake-tests+medium | --> FROM alpine /h/a/j/earthly-fake-tests+medium | [----------] 100% FROM alpine /h/a/j/earthly-fake-tests+medium...

I was using the env variables for the C version of grpc and not the go-version :facepalm: to enable grpc logging in buildkit we need: ``` global: buildkit_additional_args: ["-e", "GRPC_GO_LOG_VERBOSITY_LEVEL=99",...