earthly icon indicating copy to clipboard operation
earthly copied to clipboard

Timeout on large images

Open mhkarimi1383 opened this issue 1 year ago • 9 comments

What went wrong?

We are using earthly for PHP builds but when image get larger we will get Timeout error, and we are not able to increase timeout on push (As my searches on Code and Docs)

What should have happened?

Being able to handle larger images with increasing timeout or better pushing method

What earthly version?

earthly/earthly:v0.8.5

Buildkit Logs

No response

Other Helpful Information

SAVE IMAGE log

Error: failed to SAVE IMAGE: failed commit on ref "layer-sha256:25fa2dc62c20258c44a5423c00f1992a65685a2f000fd83c47e2f9a18f670e5f": failed to do request: Put "https://[MASKED]/v2/group/project/blobs/uploads/bbba2902-4422-45ac-8edf-4ba564fd787f?_state=qZQiifQy_YqHjygSz7DCkeZb6unneM_Y5CvA0eYxlHl7Ik5hbWUiOiJzZXJpdGEvY3JtLXNpdHJhLWFwaSIsIlVVSUQiOiJiYmJhMjkwMi00NDIyLTQ1YWMtOGVkZi00YmE1NjRmZDc4N2YiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDQtMjBUMDg6MTQ6NDAuMDE4MTc3NDQ5WiJ9&digest=sha256%!A(MISSING)25fa2dc62c20258c44a5423c00f1992a65685a2f000fd83c47e2f9a18f670e5f": net/http: timeout awaiting response headers

mhkarimi1383 avatar Apr 20 '24 08:04 mhkarimi1383

Related Builtkit issue: https://github.com/moby/buildkit/issues/4130

mhkarimi1383 avatar Apr 20 '24 09:04 mhkarimi1383

Can you clarify if this is only encountered when --push is specified? Does it occur without the --push too?

alexcb avatar Apr 22 '24 15:04 alexcb

As I remember it will not happen without --push option

mhkarimi1383 avatar Apr 22 '24 16:04 mhkarimi1383

@alexcb Any updates on this?

mhkarimi1383 avatar Jun 23 '24 14:06 mhkarimi1383

@alexcb I am also facing the same problem when pushing to ECR. However, in my case, the issue occurs even when I perform multiple pushes (images of several tens of MB in size) in a single command execution. I would like to know if there is any progress here. Or if there is a workaround, please let me know.

lomavkin avatar Aug 28 '24 07:08 lomavkin

@lomavkin For those images I'm using Dockerfile and kaniko temporarily until this issue fixes I think pushing layers one by one and ability to set timeout as a cli arg will be great

mhkarimi1383 avatar Aug 28 '24 16:08 mhkarimi1383

@mhkarimi1383

Thanks for teaching me about it. kaniko looks good.

In my case, I was failing to build and push using Earthly in GitHub Actions, so I eventually decided to build with Earthly, SAVE IMAGE, and then perform the docker push from GitHub Actions.

lomavkin avatar Aug 30 '24 18:08 lomavkin

I think Earthly is pushing all of the layers at once, So this is the actual problem I think

mhkarimi1383 avatar Aug 30 '24 18:08 mhkarimi1383

I feel the same way😅

lomavkin avatar Aug 31 '24 13:08 lomavkin