Being able to build smaller containers

[blackheaven]

I'm trying to define some simple containers:

  containers.blue = {
    name = "blue";
    copyToRoot = pkgs.buildEnv {
      name = "image-root";
      paths = [ blue ];
      pathsToLink = [ "/bin" ];
    };
    entrypoint = [ "/env/bin/blue" ];
  };

blue being a haskell.nix derivation

It creates a first layer with my derivation (~420MiB) and adds a layer with many things (coreutils-full, bashInteractive, su, etc.) which weight 15 GiB.

We should be able to passe the final derivation, or to disable extra layers.

[domenkozar]

Fully agreed, someone needs to look into container.nix and expose a knob to disable the layering.

[therealpxc]

It creates a first layer with my derivation (~420MiB) and adds a layer with many things (coreutils-full, bashInteractive, su, etc.) which weight 15 GiB.

15 GiB

I don't think those packages you name can be the culprit. They have really small closures! Together they're probably only like 100 MiB. Something weirder is going on.

[domenkozar]

https://github.com/cachix/devenv/pull/1375

[ppenguin]

I was having the same issue when trying to build a production container which includes a simple binary produced by buildGoModule. (15GB still seems like there's something else wrong too though, maybe you can get "closer to reasonable" when doing like below with the isBuilding logic)

My reference use case:

1. existing Makefile powered build of a go binary including private go module dependencies
2. desire to use devenv as a "one stop shop" to replace my own hacks and avoid implementing extending these hacks with "production image" functionality.

Immediate problems not (yet) solved with devenv:

1. Can't add binary artifacts due to a bug and the fact that you'd have to stage them (a la flake) for them to be found, which I want to avoid (no binary artifacts in git!)
2. If I use --impure, I'd expect to be able to add local relative paths anyway to e.g. copyToRoot, but it doesn't have any effect (i.e. I get a "path doesn't exist in Git repository" error)
3. Adding the artifact as a nix package is unattractive:
   1. forces me to make a nix package for my artifact
   2. which is non-trivial if you want to use buildGoModule (due to private dependency/vendoring/sandboxing nightmares => the only way I could make it work is to use go mod vendor but for this I have to add vendor to git as well)
   3. Doesn't let me use existing build processes
4. The resulting container image is "too big" and appears to contain lots of unnecessary stuff, which might be a consequence of using nix2container?

To reduce the size of the container image I already did the following:

  packages = with pkgs; lib.optionals (!config.container.isBuilding) [gnumake go-swagger gopls nix-prefetch];
  languages.go.enable = !config.container.isBuilding;
  copyToRoot = [(pkgs.callPackage ./pdnsupdate.nix {})];

which produces a 434MB image for a binary of 16MB.

If I use the above remove container tooling mod, I get a marginally better result (381MB). Note that this already uses a full nix package which I was trying to avoid in the first place, i.e. the unattractive scenario (2) mentioned above. Since (among others) I saw gcc in the container's nix-store, I'd guess that this could be the (implicit) buildInputs of the buildGoModule closure? In other words, if we use a nix package in copyToRoot, we still would need a way to only copy the runtime deps?

I'm not at all familiar with nix2container, but my experience with dockerTools.buildLayeredImage is not too bad, and it seems to allow for quite good control over the resulting image. So maybe it's worth considering going forward? As an additional alternative or as a replacement for nix2container?

Also the container functionality forces using docker, i.e. doesn't give a choice to use podman instead, which would be nice.

[therealpxc]

I think the bigger issue is that we're using the usual

which produces a 434MB image for a binary of 16MB.

If I use the above remove container tooling mod, I get a marginally better result (381MB).

Yeah, that's in line with what I expected.

I think it's clear that the main issue is not the extra tools added to the containers by default, or even the particular tools used to generate the container images. The issue is that the containers are based on the shells, but more crucially that the shells have fat closures. The shells have big closures because they're normal devShells created with Nixpkgs mkShell, so they pull in the build-time dependencies of everything in config.packages. They also pull in a whole C compiler toolchain because we're using pkgs.stdenv in the mkShell invocations by default (you can get that out of there by configuring stdenv = stdenv.noCC).

There's a WIP PR in Nixpkgs that introduces a distinction between 'build shells' (shells automatically derived from a derivation and which assume you want build-time tools associated with included packages, like the devShells produced by mkShell that we're currently using) and 'development shells', where some tools might be included in a different sort of way.

Short of waiting for that PR, depending on it now, or otherwise hoping that it meets our needs, we could choose either to handle devenv's config.packages differently (i.e., so that it is not passed directly as the packages argument of pkgs.mkShell) or to supplement it with a similar argument that means, more or less, 'packages to include in the CLI environment for runtime use but whose build dependencies are not needed'. One workaround that works for dependencies like that is just to wrap them in a buildEnv call, and include that result, rather than the package directly, into our config.packages. I did that here in order to erroneously pull the nix CLI into the environment when someone includes a Nix SCA tool by enabling the Nix language.

One consideration is that we kind of depend on the 'build shell' behavior for our interfaces for including language-specific packages-- this is why, for instance, one can just put their environment's Python libs into config.packages instead of using python.withPackages. Given that adding language-specific deps with this kind of interface is something Nix newbies often (wrongly) assume will work, e.g. with environment.systemPackages, this might be an important UX/DX feature for devenv. So maybe we don't want to change how config.packages gets plugged into mkShell but we want to introduce another class of packages that gets included more 'lightly'? There's kind of a similar question/problem for defaulting to stdenv.noCC, which might be fine for many languages but not if someone needs to compile native extensions.

Should we do an experiment against that WIP Nixpkgs PR and see (a) if, using that instead of pkgs.mkShell, we can get one of these containers with a huge closure down to a reasonable size and (b) what kind of interface is natural for specifying dependencies of different 'kinds' (i.e., those where we do care about the tools required to build against them as source code, and those where we just want to include them as a CLI tool)?

I'd be down to try that if someone wants to provide a sample project where their containers/shells normally come out bigger than necessary.

[blackheaven]

@therealpxc Thanks for your comment, I actually didn't figure out that mkShell was used to generate the image.

If no one does, I can have a try this week-end.

[domenkozar]

See https://github.com/cachix/devenv/pull/1415

[blackheaven]

Actually, IIRC, mkShell is used by default, when no entrypoint is specified.

I have made some progress, so that, when !isDev copyToRoot is passed through, fixing container size.

[ppenguin]

I have made some progress, so that, when !isDev copyToRoot is passed through, fixing container size.

Fantastic, I hope I find some time to try that sometime next week! I assume we're still "plagued" by the "problem" that we can't copyToRoot "impure" artefacts as long as they're not staged to git though, which would need to be solved too for a better UX?

[domenkozar]

I was having the same issue when trying to build a production container which includes a simple binary produced by buildGoModule. (15GB still seems like there's something else wrong too though, maybe you can get "closer to reasonable" when doing like below with the isBuilding logic)

My reference use case:

1. existing Makefile powered build of a go binary including private go module dependencies
2. desire to use devenv as a "one stop shop" to replace my own hacks and avoid implementing extending these hacks with "production image" functionality.

Immediate problems not (yet) solved with devenv:

1. Can't add binary artifacts due to a bug and the fact that you'd have to stage them (a la flake) for them to be found, which I want to avoid (no binary artifacts in git!)

2. If I use --impure, I'd expect to be able to add local relative paths anyway to e.g. copyToRoot, but it doesn't have any effect (i.e. I get a "path doesn't exist in Git repository" error)

3. Adding the artifact as a nix package is unattractive:

   1. forces me to make a nix package for my artifact
   2. which is non-trivial if you want to use buildGoModule (due to private dependency/vendoring/sandboxing nightmares => the only way I could make it work is to use go mod vendor but for this I have to add vendor to git as well)
   3. Doesn't let me use existing build processes

4. The resulting container image is "too big" and appears to contain lots of unnecessary stuff, which might be a consequence of using nix2container?

To reduce the size of the container image I already did the following:

packages = with pkgs; lib.optionals (!config.container.isBuilding) [gnumake go-swagger gopls nix-prefetch]; languages.go.enable = !config.container.isBuilding; copyToRoot = [(pkgs.callPackage ./pdnsupdate.nix {})]; which produces a 434MB image for a binary of 16MB.

If I use the above remove container tooling mod, I get a marginally better result (381MB). Note that this already uses a full nix package which I was trying to avoid in the first place, i.e. the unattractive scenario (2) mentioned above. Since (among others) I saw gcc in the container's nix-store, I'd guess that this could be the (implicit) buildInputs of the buildGoModule closure? In other words, if we use a nix package in copyToRoot, we still would need a way to only copy the runtime deps?

I'm not at all familiar with nix2container, but my experience with dockerTools.buildLayeredImage is not too bad, and it seems to allow for quite good control over the resulting image. So maybe it's worth considering going forward? As an additional alternative or as a replacement for nix2container?

Also the container functionality forces using docker, i.e. doesn't give a choice to use podman instead, which would be nice.

We should build that container only for shell container, and for the rest we shouldn't.

[ppenguin]

We should build that container only for shell container, and for the rest we shouldn't.

I'm not really sure what you mean? That devenv should only support building containers with the development environment and not "runnable artefact containers"?

(In that case I'd sure like to plead for the latter functionality, if not integrated in the original implementation then at least as an addition.)

[dcdq]

Not sure if this is still going but I actually talked to one of the cachix team in discord a year and a half back about this. I had a nix2container implementation for a pure esm bun typescript monorepo. It was down to 54mb for the entire codebase (non compiled apis and 1.5mb of bundled expo for browser). At the time 52mb was the bun binary. Has been running in production since with no issues, but just enough friction in deployments to make it annoying and not enough to actually fix it all this time.

This was the issue that kept me from sticking with devenv that time beyond the first day trying. After getting cosy in devenv and laying down some patterns, the container spat out at the end was >10GB, completely out of the left field based on how it went up till that point. Couple of days ago, just started basing a new monorepo design on devenv again, and the progress has been absolutely awesome in every regard aside from that.

All that is to say, I think it would be odd to have container functionality in devenv that targets the shell. The shell can be ran by any human and machine in a much more deterministic way, out of the box, with granular cache and no extra hoops. The only thing most people should expect upon seeing devenv's container support after skimming the docs sidebar is that this offers a more indispensable advantage vs maintaining own flakes - being able to go to production with no friction.

Will just drop this here https://github.com/pdtpartners/nix-snapshotter, which might be using nix2container at least in part, for reference. Circling back to this whole issue now with the aim of getting it all into kube this way, but it would be a lot better if devenv focused on parity with production. Worst case scenario, is at least riding the same inputs from devenv into the container build process. If devenv provided some libs to help some of the pinning and hash busting in a structured way (through scripts/tasks, which are awesome btw), or any other new patterns that have emerged in the last couple of years, it would already be more compelling and could just as well provide for an overall simpler generic solution here for people to compose their process rather than a one-size-fits-all that doesn't actually fit anybody.

My 2c for what it's worth, I'll drop back here or into discord if I find anything more useful than just for my project this time around.