Push to harbor fails with "413 Request Entity Too Large"

[ligouras]

I'm running a bitnami/harbor stack behind nginx-proxy.

I have a github action that uses build-push-action and up until now it was working just fine. It suddenly failed with the following error message:

#21 exporting to image
#21 exporting layers
#21 exporting layers 15.3s done
#21 exporting manifest sha256:ba5abdea2753fda52ff022e022195f803fc9a43eaddb21fd2b355d08cba86cca done
#21 exporting config sha256:159aa8a6470b5fcc026f7091c7e94f7e2fb36256badec6548d06a1a7ea03941b done
#21 pushing layers
#21 ...

#21 exporting to image
#21 pushing layers 1.2s done
#21 ERROR: failed to copy: unexpected status: 413 Request Entity Too Large

nginx logs do not show a 413, rather the PUT request which uploads the layer returned 201 Created:

"PUT /v2/myproject/blobs/uploads/f3b7a527-750c-4004-a5de-66cbf1daa04c?_state=P6j...iJ9&digest=sha256%3Af7a1b93d64e8fed378f0f8148c042e4329fa7a353e2991af33390eb6c01338ba HTTP/1.1" 201 0 "-" "buildkit/v0.10"

I went so far as to capture the traffic with wireshark and I can see packets marked as [TCP WIndow Full] and [TCP Zerowindow] in that PUT request's stream. If this is network related, why does the server reply with 201, and why does the client report 413 ?

FYI: nginx is configured with client_max_body_size 0; and when I manually build and push using docker cli, everything works just fine. Other repos where I use build-push-action also work just fine.

Any hints towards a solution are greatly appreciated!

[crazy-max]

Is the client_max_body_size directive also applied to the http context or just server? Can you also post your BuildKit logs please?

[ligouras]

Thanks for the reply, I added buildkitd-flags: --debug to setup-buildx-action step, but I didn't get much meaningful information tbh, but ... I think I just found the culprit!

The harbor domain was proxied through Cloudflare, disabling it finally let the action push the image successfully. I guess it make sense for CF to return 413 because I never got it in my nginx logs. It feels weird though, why did building and pushing through the cli succeed? Did I just stumble into an edge case where manual build produced a layer within CF's limits while the github builder went over?

Either way thanks for taking the time to reply, I have one final question, is it possible to somehow override DNS, setting like a custom hosts directive for buildx that will make it bypass CF? Or is the only viable option disabling CF proxying altogether?

Cheers!