Buildx memory leak

[ifatregev]

We are experiencing a memory issue with Buildx on our CI pipeline instances, might be related to this issue: https://github.com/moby/buildkit/issues/1726

We are reusing a specific machine for several builds, and the second build inherits the "high memory" from the previous build on the same machine.

Here are some metrics:

We are using these commands at the beginning of each pipeline: docker buildx install docker buildx create --use echo 'FROM scratch' | docker buildx build -

Would appreciate your suggestions to resolve this issue 🙏

[tonistiigi]

Do you have specific reproduction steps? Seems to maintain stable ceiling to me once the node goes under memory pressure.

[ifatregev]

We are using Buildx in our massive and complicated CI pipeline flow, therefore I can’t add reproducing steps (other then the Buildx commands I attached above). Generally in our pipe, we install Buildx and creating a builder and using it, at early stages of the pipe. We build our services using docker buildx bake while using a remote registry for caching. Then we are running our services using docker-compose up, and running our e2e tests against them.

[zjwon]

I had the same problem with buidctl, k8s statefulset.rootless. The pod was restarted due to memory overflow

buildctl \
  --addr kube-pod://buildkitd-0 \
  build \
  --frontend dockerfile.v0 \
  --local context=. \
  --local dockerfile=. \
  --output type=image,name=example,push=true

[maximegaillard]

Same here. I tried to run buildkit on gitlab runners on k8s (currently on kaniko). OOMkilled each time, even with bigger nodes (started with 4Gb of memory to 18Gb).

I had trouble to debug it and find out what's going on... If someone has an idea...

[peraronsson]

If you repeatedly build a JS/node project using node:alpine for example, it will eventually run out of memory. The first 10 times might work fine, but seems like memory usage builds up to eventually crash the build process. Increasing the amount of available memory will just delay the inevitable. This seems like a memory leak.

[jsternberg]

We recently merged a change that may be related to this issue. If you are still experiencing this issue, do you think you can try building buildkit off of master and see if it reduces the amount of memory that is being allocated? The issue we resolved could manifest itself to seem like a memory leak because of the large number of accidental allocations. For some builds, it could result in the memory remaining in use for the duration of the build and for repeated builds Go will sometimes be too slow to free the memory and the Linux OOM killer will kill the process before the garbage collector runs.

If that change doesn't resolve this issue and it's still occurring, I can try to prepare some instructions for how to obtain a heap profile. That will help us narrow down exactly what is happening.

[jsternberg]

I'm going to close this issue for now because of the PR here: https://github.com/moby/buildkit/pull/4116

If you're still running into buildkit using too much memory after testing with that change, we'd really like to hear about it and I can reopen this issue to continue with further investigation. Thank you for taking the time to open this issue and help bring this to our attention.