Limit used memory/CPU or parallel jobs

[WolfspiritM]

Hi,

I tried to use buildx to speed up our pipeline. We have a huge docker-compose with many services that need to be build and our buildserver is a bit limited with memory and cpu.

Once I started using buildx to build our project with docker buildx bake -f docker-compose.yml after a while the buildserver stopped responding and it was out of memory and cpu. It had many many build processes running in parallel.

Is there any way to limit the amount of ram and cpu used or at least specify the maximum allowed parallel jobs? Right now it seems to run everything at once which seems to overwhelm our buildserver.

[Aposhian]

This should be as simple as passing on runtime options to the buildkit container, right?

[Aposhian]

Also, docker buildx build already supports --memory and other throttling flags. Mind if I make a PR to add those same options to buildx bake? The only issue is still specifying the number of parallel jobs, since simply passing on --memory would give that much to each build job, right?

[MrHash]

In my testing setting CARGO_NET_GIT_FETCH_WITH_CLI=true resolves the OOM 137 error code during the cargo crate fetching stage using buildx

[tnaroska]

interested in this feature as well. There is max-parallelism option in buildkit.toml. However that has no effect on operations from individual RUN steps in the dockerfile.

It would be more useful to have options to limit the cpu/memory when creating a docker-container instance using buildx create

[aidapsibr]

interested in this feature as well. There is max-parallelism option in buildkit.toml. However that has no effect on operations from individual RUN steps in the dockerfile.

It would be more useful to have options to limit the cpu/memory when creating a docker-container instance using buildx create

I'm running into similar issues with the Kubernetes driver. If I limit the CPU I just end up CPU throttled. Seems like my build jobs aren't fanning out when I set replicas either, so if I bake too many images at once I just eventually get timeouts. I add this because the same issue could come up with the docker-container solution with that solution alone.

[crag-h4k]

In my testing setting CARGO_NET_GIT_FETCH_WITH_CLI=true resolves the OOM 137 error code during the cargo crate fetching stage using buildx

I'm still running into this same issue :(

[nicks]

Compose supports COMPOSE_PARALLEL_LIMIT. Maybe it would be good if buildx bake respected this as well?

https://docs.docker.com/compose/environment-variables/envvars/#compose_parallel_limit

Alternatively, maybe add a BUILDX_PARALLEL_LIMIT to match the compose CLI?

[Borda]

Also, docker buildx build already supports --memory and other throttling flags. Mind if I make a PR to add those same options to buildx bake?

That would be great!