buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard
verified publisher icon moby

gha cache remote: error writing layer blob with batch checking

Open crazy-max opened this issue 1 year ago • 2 comments

This happens since we enabled support for batch checking of existing keys in build-push-action 6.4.0: https://github.com/docker/build-push-action/issues/1181

ERROR: failed to solve: error writing layer blob: Patch "https://acghubeus1.actions.githubusercontent.com/Odv8msvJgMS5iWxgU43K2EnWLcwj1p7E8K5nLVIfXGs2O9tj1t/_apis/artifactcache/caches/205053": read tcp 172.17.0.2:57224->20.60.63.33:443: use of closed network connection

Other report but this time http: read on closed response body:

2024-07-16T17:00:58.9862132Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A25749db20348e6351d5f6f0377fd90683b7c9c38cb8c5e8828b64418414f9d2d&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9868519Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A3713021b02770a720dea9b54c03d0ed83e03a2ef5dce2898c56a327fee9a8bca&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9875706Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ac205e3cd6d4fe77e75ac773d2095d5b077da4e409208e91a437736ac3516bf8a&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9881773Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A9d979625e03710120862618df652a44147c39cea63f81fc3317150576d13970d&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9887767Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A5e1d8a175912f23735362c719758739f9820779a4c3e65183d14a78a62c4d385&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9894218Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ac1e08517414cd778c7f615a79a994f56eb3cd786e6a1d18327cfc33033d37fb7&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9900522Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Afd5a4f3a56d5413c8dac531df69282453b580858b02db963336b39e7e13fb2fc&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:58.9907061Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A9674d23d14435baed80cb7b4d2f79eed4fd8a7b170e1cd9c904cdabc14c478d5&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0027843Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A740a1bf374b050206a2e7dbebe3fc39f28b07ad5a7d3ff01ca94e381633f93c1&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0034235Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A334a3bdc2a7bbe4ddc90029152da641bbba2e051aa401859fb9ec6ee84589279&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0040924Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A4ce000a43472e4a2527834764b5044674760f1e2a766480798d03a93b51a0b39&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0047214Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Ab4ea33a8f66c0b2df9875ba27c31925f1981f302f57fc9dfee62449cb5f23b3c&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0053463Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A34ce04bd839bdd14be00b3ecc06ab94cd6a74063954fe7007d30f1b3607efead&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0059767Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3A09ac36fdd82d2b54719ef1b6d4b31b34e8e4a1000cb4ffad9dae86c8a0b17db6&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0066136Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Af836a8b5317b9de84ae0e8852420f7067e99c33a94281958bc1be35e106d18de&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0072403Z time="2024-07-16T17:00:49Z" level=debug msg="load cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/cache?keys=buildkit-blob-1-sha256%3Aa4d2191ac6530b4133830d8ca3369ea70c6382652778b846329993ed82ffd279&version=693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2"
2024-07-16T17:00:59.0086340Z time="2024-07-16T17:00:50Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:09ac36fdd82d2b54719ef1b6d4b31b34e8e4a1000cb4ffad9dae86c8a0b17db6\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0090243Z time="2024-07-16T17:00:50Z" level=debug msg="save cache resp: {\"cacheId\":1421}"
2024-07-16T17:00:59.0097314Z time="2024-07-16T17:00:50Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1421, range 0-237"
2024-07-16T17:00:59.0135751Z time="2024-07-16T17:00:50Z" level=debug msg="commit cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1421, size 238"
2024-07-16T17:00:59.0146114Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:18ebc6ff070748ff10785729f9b5b7a953c708f6c25b5ff759cd83136eab965b\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0149916Z time="2024-07-16T17:00:51Z" level=debug msg="save cache resp: {\"cacheId\":1422}"
2024-07-16T17:00:59.0152425Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1422, range 0-1536"
2024-07-16T17:00:59.0166115Z time="2024-07-16T17:00:51Z" level=debug msg="commit cache https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/1422, size 1537"
2024-07-16T17:00:59.0174606Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:25749db20348e6351d5f6f0377fd90683b7c9c38cb8c5e8828b64418414f9d2d\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0180806Z time="2024-07-16T17:00:51Z" level=debug msg="save cache req https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches body={\"key\":\"buildkit-blob-1-sha256:277e32004e30b487b4bd089b32d8b35c1698a9379a81a4bb5bab75b16554457e\",\"version\":\"693bb7016429d80366022f036f84856888c9f13e00145f5f6f4dce303a38d6f2\"}"
2024-07-16T17:00:59.0184251Z time="2024-07-16T17:00:51Z" level=debug msg="save cache resp: {\"cacheId\":***}"
2024-07-16T17:00:59.0186849Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***, range 0-33554431"
2024-07-16T17:00:59.0197410Z time="2024-07-16T17:00:51Z" level=debug msg="upload cache chunk https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***, range 33554432-51895362"
2024-07-16T17:00:59.0203898Z time="2024-07-16T17:00:53Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing layer blob: Patch \"https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***\": http: read on closed response body"
2024-07-16T17:00:59.0213290Z error writing layer blob: Patch "https://acghubeus2.actions.githubusercontent.com/***/_apis/artifactcache/caches/***": http: read on closed response body
2024-07-16T17:00:59.0216172Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0217227Z github.com/tonistiigi/go-actions-cache.(*Cache).doWithRetries
2024-07-16T17:00:59.0218231Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:455
2024-07-16T17:00:59.0219446Z github.com/tonistiigi/go-actions-cache.(*Cache).uploadChunk
2024-07-16T17:00:59.0220405Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:410
2024-07-16T17:00:59.0221346Z github.com/tonistiigi/go-actions-cache.(*Cache).upload.func1
2024-07-16T17:00:59.0222352Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:314
2024-07-16T17:00:59.0223168Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0223919Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0224582Z runtime.goexit
2024-07-16T17:00:59.0225059Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0225495Z 
2024-07-16T17:00:59.0226022Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0227023Z github.com/tonistiigi/go-actions-cache.(*Cache).uploadChunk
2024-07-16T17:00:59.0228035Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:412
2024-07-16T17:00:59.0229033Z github.com/tonistiigi/go-actions-cache.(*Cache).upload.func1
2024-07-16T17:00:59.0230129Z 	/src/vendor/github.com/tonistiigi/go-actions-cache/cache.go:314
2024-07-16T17:00:59.0231020Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0231800Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0232506Z runtime.goexit
2024-07-16T17:00:59.0232985Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0233436Z 
2024-07-16T17:00:59.0233984Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0235383Z github.com/moby/buildkit/cache/remotecache/gha.(*exporter).Finalize
2024-07-16T17:00:59.0236295Z 	/src/cache/remotecache/gha/gha.go:230
2024-07-16T17:00:59.0237170Z github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
2024-07-16T17:00:59.0238135Z 	/src/solver/llbsolver/solver.go:693
2024-07-16T17:00:59.0239006Z github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
2024-07-16T17:00:59.0239929Z 	/src/solver/llbsolver/solver.go:1064
2024-07-16T17:00:59.0240634Z github.com/moby/buildkit/solver.(*Job).InContext
2024-07-16T17:00:59.0241322Z 	/src/solver/jobs.go:808
2024-07-16T17:00:59.0242035Z github.com/moby/buildkit/solver/llbsolver.inBuilderContext
2024-07-16T17:00:59.0242874Z 	/src/solver/llbsolver/solver.go:1060
2024-07-16T17:00:59.0243741Z github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1
2024-07-16T17:00:59.0244655Z 	/src/solver/llbsolver/solver.go:674
2024-07-16T17:00:59.0245304Z golang.org/x/sync/errgroup.(*Group).Go.func1
2024-07-16T17:00:59.0246047Z 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
2024-07-16T17:00:59.0246717Z runtime.goexit
2024-07-16T17:00:59.0247160Z 	/usr/local/go/src/runtime/asm_amd64.s:1650
2024-07-16T17:00:59.0247594Z 
2024-07-16T17:00:59.0248146Z 7 v0.14.1 buildkitd --debug --allow-insecure-entitlement=network.host
2024-07-16T17:00:59.0248944Z main.unaryInterceptor
2024-07-16T17:00:59.0249401Z 	/src/cmd/buildkitd/main.go:716
2024-07-16T17:00:59.0250413Z google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
2024-07-16T17:00:59.0251587Z 	/src/vendor/google.golang.org/grpc/server.go:1154
2024-07-16T17:00:59.0252524Z github.com/moby/buildkit/api/services/control._Control_Solve_Handler
2024-07-16T17:00:59.0253429Z 	/src/api/services/control/control.pb.go:2466
2024-07-16T17:00:59.0254130Z google.golang.org/grpc.(*Server).processUnaryRPC
2024-07-16T17:00:59.0254895Z 	/src/vendor/google.golang.org/grpc/server.go:1343
2024-07-16T17:00:59.0255615Z google.golang.org/grpc.(*Server).handleStream
2024-07-16T17:00:59.0256337Z 	/src/vendor/google.golang.org/grpc/server.go:1737
2024-07-16T17:00:59.0257120Z google.golang.org/grpc.(*Server).serveStreams.func1.1
2024-07-16T17:00:59.0258229Z 	/src/vendor/google.golang.org/grpc/server.go:986
2024-07-16T17:00:59.0258918Z runtime.goexit
2024-07-16T17:00:59.0259416Z 	/usr/local/go/src/runtime/asm_amd64.s:1650

crazy-max avatar Jul 17 '24 09:07 crazy-max

@crazy-max Do you have reproduction or any follow-up for this issue?

tonistiigi avatar Oct 15 '24 23:10 tonistiigi

Was not able to repro since we added batch requests in BuildKit CI pipeline last year. Without enabling it by default in our GHA it's a bit hard to have more reports. Do you think we can try to enable it back by default? (https://github.com/docker/build-push-action/pull/1183)

crazy-max avatar Aug 26 '25 12:08 crazy-max