Ben Davies

@dhadka thanks for the reply and attention. step debug logging enabled for (although it doesn't provide much more info here) run 3: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/3 run 4: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/4 run 3 was very...

good spot @dhadka the slow step in those runs is `preparing build cache for export` which correlates to the `remove snapshot` and following lines in the buildkit logs that you've...

btw, just pushed a cache busting commit: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2163336573 (the build was quicker, but still very slow.) and there are some real long delays for: 1. fetching manifest 2. between 2...

i should also note that, to work around this, we have currently commented out the `cache-to` option on all our image builds. This "works", i.e. we don't get hangs any...

@tonistiigi I did 400 layers as i was trying to reproduce the cache throttling behaviour that I thought we were seeing in our main private repo. We are seeing the...

another really long delay in the same job: `docker/build-and-push` logs: ``` 2022-04-12T11:32:09.8431241Z #15 importing cache manifest from gha:1241927905649341978 2022-04-12T11:35:47.8803590Z #15 DONE 259.0s ``` `buildkit` logs for roughly the same timestamp:...

@dhadka that would be very much appreciated! Like is said in the OP, we are hitting these limits (i think) pretty constatly (before i disabled cache writing) with a pretty...

i'll give it a go

@tonistiigi seems correct. local cache is also slow (but not as slow) run 1 (unprimed local cache): https://github.com/bendavies/docker-build-cache-limits/actions/runs/2167840770/attempts/1 run 2: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2167840770/attempts/2 the slow bit in the local cache logs: `docker/build-and-push`:...

i'm seeing a few of these 401 Unauthorized, but not sure if it's related or even a problem: ``` time="2022-04-14T14:50:38Z" level=debug msg="fetch response received" digest="sha256:c42883e97de28a9f7e297275b9c8cdee26a80386f550092fb07e2929a52e7752" mediatype=application/vnd.docker.container.image.v1+json response.header.content-length=73 response.header.content-type=application/json response.header.date="Thu, 14...