Allow a way to use buildLayeredImage

[andr-ec]

What problem are you trying to solve?

I want to be able to create docker images that are deterministic and reduce the amount of layers that need to be sent to the registry. similar to how is explained in this article: https://xeiaso.net/talks/2024/nix-docker-build/

What solution would you like?

Ideally a command that builds the docker image that is ready to be uploaded to a registry. but it would also be ok if it output a nix file configured with all the existing inputs that has buildLayeredImage implemented.

Since cross compilation is hard it would be fine if it only worked on linux since in our use case it would only be run in ci.

Alternatives you've considered

The current docker generate works but isn't as minimal or as optimal as using buildLayeredImage directly with nix.

[mohsenari]

I saw that and it seems like a very good idea. I need to do some testing to make sure an image built on MacOS can be used in Linux. Or if not what is the workaround.

[nhuray]

I have exactly the same need.

I would like to build a docker image defining a single layer with all the devbox dependencies.

This base image would be used to build other docker images in a multi-stage build. That base image might also be used in Github action as a container as an alternative to the devbox-install-action.

[apgrucza]

Hopefully this would result in smaller Docker images too, because I'm finding my CI takes a very long time to pull large images produced via devbox generate dockerfile.

[drewgingerich]

I think this would be an amazing feature, and would skyrocket the value of Devbox because it could become a tool not just for development but also for creating top-notch production images.

For example to create production images at work we currently have to choose between:

1. Official images, often with bloat containing vulnerabilities with no fix version
2. Distroless images which are super slim but a pain to work with
3. Nix which works great but is an extremely hard sell

Devbox providing a friendly layer over Nix would be the ticket out of this mess. Obviously Devbox is already doing this for dev environments.

@nhuray

I would like to build a docker image defining a single layer with all the devbox dependencies.

My understanding is that a major advantage of Nix-built Docker images is that every dependency is a separate layer, which provides maximal caching/deduplication in the registry. That said, I'll take anything right now if it leads to a minimal production image!

[andr-ec]

Hey guys I've been thinking about this and I found a decent work around.

Basically you can base your dockerfile around Nixery using the devbox.lock commits for each package.

The start of the dockerfile would look like this:

# Stage 1: Main tools
FROM nixery.dev/shell/beam28packages.elixir/beam28packages.hex/beam28packages.rebar3/gcc/glibc.dev/cacert/curl/gnumake AS base

# Stage 2: Node.js with specific commit
FROM nixery.dev/nodejs:076e8c6678d8c54204abcb4b1b14c366835a58bb AS nodejs-layer

FROM base

# Copy nix store from Node.js
COPY --from=nodejs-layer /nix /nix

# Make Node.js
RUN ln -sf /nix/store/*nodejs*/bin/* /bin/ 2>/dev/null || true

WORKDIR /app

From there you just modify the dockerfile like any other dockerfile.

It's loaded using https://nixery.dev/ You'd probably want to host your own version of nixery.

I created a shell script that does it for you. I'm sure it could easily be integrated into devbox.

A shell script that generates a dockerfile from a devbox.json and devbox.lock using nixery.dev

I still need to test it more but hope it helps!

[apgrucza]

@andr-ec thanks for sharing this. A quick test shows it reduces my image size (which used devbox install and Nix garbage collection) from 1.39GB to 1.13GB, which is a decent improvement. But after adding in Devbox and Nix, as well as the dependencies in the Devbox image (binutils git xz-utils wget sudo), the size goes up to 1.88GB.

When nixery.dev builds the entire image itself (including the extra packages above), it's only 882MB. Unfortunately this is not practical for Devbox environments because nixery seems to be limited to using a single nixpkgs commit, which explains why you took the approach that you did.

Did you try using buildLayeredImage at all (as mentioned in your original post)? It would be interesting to see if it's able to produce images closer to the size that nixery.dev does.

[apgrucza]

I just tried buildLayeredImage and got a size of 750MB, which is even better. But the problem with both the nixery and buildLayeredImage approaches is that when you run devbox shell in the container, Devbox insists on installing all your packages again, even though the store paths in devbox.lock already exist. It adds a lot of additional packages to /nix/store that don't seem to be necessary.

If you use --recompute=false, then Devbox doesn't try to install those extra packages and things seem to work okay, but --recompute=false would prevent certain Devbox features from working, such as the auto-patching of native libraries.

It would be good to understand why Devbox wants to reinstall the packages.

[andr-ec]

Interesting, In my dockerfile I don't add devbox again. I'm just using devbox as a package manager in dev so adding it again in wouldn't help in my use case. But maybe others do need that.

I'm sure nixery could be more efficient if it supported multiple commits. That might be interesting to look at next.

[apgrucza]

I initially left Devbox out of the image but later realised I need it as I'm reliant on its --patch feature. Less critically, I also use the init_hook and scripts attributes in devbox.json.

I discovered that if I copy /nix/var/nix/db into the image, the "Ensuring packages are installed" stage is quick and Devbox doesn't reinstall the packages. But then in the "Computing the Devbox environment" stage it still downloads a large number of unnecessary packages, so I reverted that.

To ensure patched packages were included in the image, I ended up letting Devbox install and patch all the packages it wants to in a clean build environment and then created an image by passing specific Nix store paths into streamLayeredImage. I only passed store paths from PATH, devbox.lock, and the symlink targets in .devbox/nix/profile/default/bin (to capture patched packages). I also included Devbox and Nix (required by Devbox).

With this image, --recompute=false must always be used to prevent Devbox from downloading extra packages. The final image ended up being 1.7GB (uncompressed), compared to 2.07GB when using devbox generate dockerfile. (Removing Devbox and Nix would reduce that by another 92MB.)

It's not working flawlessly though. The Python plugin gives a warning (which I can work around with touch .devbox/venv_check_completed, but I get a cannot execute: required file not found when running ruff which I haven't gotten to the bottom of yet. It feels a bit fragile, and there may be other issues I've not found yet. At this stage, I'm not sure whether the benefits of a reduced image size and layer-caching outweigh the risk of problems occurring in a "hacked" Devbox environment.