buildkit icon indicating copy to clipboard operation
buildkit copied to clipboard

Allow ignoring errors when exporting remote cache

Open tonistiigi opened this issue 3 years ago • 3 comments

previous issue https://github.com/docker/buildx/issues/786 related https://github.com/docker/build-push-action/issues/515

Cache export should take an option for controlling what to do when cache exporting fails. There are cases where there are temporary issues with the cache backend and the user would like it to not fail the build.

tonistiigi avatar Jan 25 '22 03:01 tonistiigi

For reference, I am posting what I posted on the previous issue here.

I know this issue is closed, but I would like to bring up that GitHub Support is claiming these are timeouts on their API. Their recommendation is to patch this code to retry on 503s, with the backoff strategy.

While that kind of patch would probably resolve this for more cases than not, it still leaves a possibility that the builds will fail. I am trying to push back and have them provide more information on why these requests are timing out as, to me, that is a GitHub Infrastructure issue, not OSS issue.

ellisio avatar Mar 01 '22 16:03 ellisio

GitHub support has requested that anyone experiencing this issue, please open a support ticket and reference ticket #1469808. They are wanting to build a case around this for prioritization for some infrastructure changes involving the caching layer.

ellisio avatar Mar 10 '22 16:03 ellisio

Hitting an issue about half of the time when using the local cache. Would be super useful to gracefully ignore a failure when writing to the cache.

#35 [app] exporting cache
#35 preparing build cache for export
#35 preparing build cache for export 0.4s done
#35 writing layer sha256:000f02d5bab2e1769bf724dc6a99cf1882267577981c5eac519ac786ad88da57 done
#35 writing layer sha256:0ab5ec7719940ff567e31ea7bcb202c5a70cf97551c77fd8a9ffcffe8d8c6a57 0.1s done
#35 writing layer sha256:1acf565088aae3ef2159885f29853bce88eb16082b0c98fcacd08fc9008c84b9
#35 writing layer sha256:1acf565088aae3ef2159885f29853bce88eb16082b0c98fcacd08fc9008c84b9 0.1s done
#35 writing layer sha256:1c6f497c8d6139eaf9da229089ce3330071db73cef17ebedcd933ae5b02d0e49
#35 writing layer sha256:1c6f497c8d6139eaf9da229089ce3330071db73cef17ebedcd933ae5b02d0e49 2.9s done
#35 writing layer sha256:37219ced0f729d39f75fe142ea182463cd5a1ec7cd8666319264523093ad7432 done
#35 writing layer sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51
#35 writing layer sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51 1.0s done
#35 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#35 writing layer sha256:5084fa7ebd744165b15df008a9c14db7fc3d6af34cce64ba85bbaa348af594a3 done
#35 ERROR: error writing layer blob: Canceled: grpc: the client connection is closing: unknown
------
 > [app] exporting cache:
------
error: failed to solve: error writing layer blob: Canceled: grpc: the client connection is closing: unknown
Docker buildx bake failed. Please make sure docker is running.
Command failed with exit code 1: docker bake -f /home/runner/docker-compose.yml --push
    Error: Command failed with exit code 1: docker 
     bake -f
     /home/runner/docker-compose.yml --push
Error: Process completed with exit code 2.

tjhiggins avatar Jul 14 '22 00:07 tjhiggins