Exporting build cache within Github Actions sometimes fails with "Canceled desc = context canceled"

[sbrocket]

An example failure: https://github.com/sbrocket/failsafe/runs/3236780733?check_suite_focus=true#step:7:730

#26 exporting cache
#26 preparing build cache for export
#26 preparing build cache for export 76.5s done
#26 writing layer sha256:0656420994bf7246e29bdbab891c129c296686d93a55bf9c133bb1f9382179fc
#26 writing layer sha256:0656420994bf7246e29bdbab891c129c296686d93a55bf9c133bb1f9382179fc 0.1s done
#26 writing layer sha256:181a01136f0a651acb1e2523a9967c75d38343c157ba7ac696dea06cd8bfb664
#26 writing layer sha256:181a01136f0a651acb1e2523a9967c75d38343c157ba7ac696dea06cd8bfb664 0.3s done
#26 writing layer sha256:3cb092620ab2812c5de7119de04b8fc9d8fea9d886ccc3f9450e4ed408e74950 done
#26 writing layer sha256:42bc939754ba8b2c28bf452f09cc99d25424a737e110c2dc9ac63ea00b1d5844
#26 writing layer sha256:42bc939754ba8b2c28bf452f09cc99d25424a737e110c2dc9ac63ea00b1d5844 4.6s done
#26 writing layer sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba 0.1s done
#26 writing layer sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b
#26 writing layer sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b 0.1s done
#26 writing layer sha256:5c22c6ce94671cbed201daa9a5b2041d8af4105ae5fab8f026f39b827c404749
#26 writing layer sha256:5c22c6ce94671cbed201daa9a5b2041d8af4105ae5fab8f026f39b827c404749 4.7s done
#26 writing layer sha256:7dcffe0848611a8d86e4514ce7d8022922a9296f13373bba76c8d426b51d1aee
#26 writing layer sha256:7dcffe0848611a8d86e4514ce7d8022922a9296f13373bba76c8d426b51d1aee 8.3s done
#26 writing layer sha256:97954ac451074cec19160fcbc2082d1ec75ad151d383b31799021ab843375040
#26 writing layer sha256:97954ac451074cec19160fcbc2082d1ec75ad151d383b31799021ab843375040 0.1s done
#26 writing layer sha256:a35c9a96f241161b00295512710e1d3e8629a59275e14b086cc2b3f1686587e0
#26 writing layer sha256:a35c9a96f241161b00295512710e1d3e8629a59275e14b086cc2b3f1686587e0 0.0s done
#26 writing layer sha256:acb6d8862af2c24e1a198aa006eac52703f2198166e6b89b870b2155082ac8c0 done
#26 writing layer sha256:bcdb8fc99176d427bd7515e167ad82009ca0c7788fc23e1651dc09975abe2777 done
#26 writing layer sha256:c0be8eca5527a01ceab7f038cf1613c804e74597a07cde2e89d3c7bb2bc3d952
#26 writing layer sha256:c0be8eca5527a01ceab7f038cf1613c804e74597a07cde2e89d3c7bb2bc3d952 1.0s done
#26 writing layer sha256:dbce84d8b90c33af9caebf171caee142c5f22c22bbbef3deeaac52eaa2ee4d4c
time="2021-08-04T01:31:30Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected="sha256:eddc66dd44315d4f6e64aaa7a33ecd7e2c5117b69890a81ab5dbfb9792319ee4" ref="sha256:eddc66dd44315d4f6e64aaa7a33ecd7e2c5117b69890a81ab5dbfb9792319ee4" total=4871
#26 writing layer sha256:dbce84d8b90c33af9caebf171caee142c5f22c22bbbef3deeaac52eaa2ee4d4c 3.7s done
#26 writing config sha256:d8a1be81cc7b682cb319ae280d66c32c0d9bb7ed78ad923fd18cf3f3ba2e701d done
#26 writing manifest sha256:eddc66dd44315d4f6e64aaa7a33ecd7e2c5117b69890a81ab5dbfb9792319ee4 done
#26 DONE 99.7s

I saw #369 which looks like the same issue, but it seems like the issue was not fully fixed by whatever fixes were applied last December.

Workflow: https://github.com/sbrocket/failsafe/blob/59cc04e/.github/workflows/docker-build.yml Dockerfile: https://github.com/sbrocket/failsafe/blob/59cc04e/Dockerfile

[crazy-max]

Should have been fixed in https://github.com/containerd/containerd/pull/5227 but looks like it still occurs with BuildKit 0.9.0 (containerd 1.5.3). cc @tonistiigi @AkihiroSuda.

[tonistiigi]

another fix https://github.com/containerd/containerd/pull/6145. Looks like the race just got a lot narrower with the previous one.

Note that this is not an actual error, just an invalid log message. Your build is not marked as fail and the cache export is not corrupted.