devenv icon indicating copy to clipboard operation
devenv copied to clipboard
verified publisher icon cachix

Container build fails if build from within shell container

Open rob-mur opened this issue 3 months ago • 8 comments

Describe the bug

If you build a shell container and then from within that container you try to build a container you get a crash about devenv-container-etc.

It would be beneficial to allow this so we could cache our shell setup into an image for use in CI, but still be able to build containers from within that image.

I'm unsure if this is a limitation of nix2container or devenv so I've posted in both.

For now from what I can manage to debug, the current options are:

  • Use a dockerfile in CI when building production images (not ideal, as means we have a different system locally to CI)
  • dont use the base image in CI (slow builds as it needs to build the shell first)

To reproduce

  • Gist with minimal setup
  • devenv container --registry docker-daemon: copy shell to build the container
  • docker run shell devenv container build -v shell to get verbose logs of the error

The crash:

because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/856i1ajaci3kmmp15rifacfz3jvn5l3q-patch-2.8 because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/6yjb3zdj448rm8qsmpiq3f67kvj5683a-bzip2-1.0.8-bin because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/5yzw0vhkyszf2d179m0qfkgxmp5wjjx4-move-docs.sh because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/0y5xmdb7qfvimjwbq7ibg1xdgkgjwqng-no-broken-symlinks.sh because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/03nvbw411p097h6yxjghc33rbcrjfb9d-gawk-5.3.2 because already present in a parent layer
[36mINFO[0m[0000] Excluding path /nix/store/8yk6k80h03hz8a6743x53xqlcdpafda8-stdenv-linux because already present in a parent layer
Failed accessing path "/nix/store/si43v2qd4yl8if9ibwcxnl9rdx0h1c6x-devenv-container-etc": lstat /nix/store/si43v2qd4yl8if9ibwcxnl9rdx0h1c6x-devenv-container-etc: no such file or directory
[31;1merror:[0m
       [31;1merror:[0m Cannot build '[35;1m/nix/store/jpm5hivrp8vs2rgy6j9l5vmp1k1j7pv5-layers.json.drv[0m'.
       Reason: [31;1mbuilder failed with exit code 1[0m.
       Output paths:
         [35;1m/nix/store/4p2mglzmzc9vspx7c3m4kl335sblfrgf-layers.json[0m
[31;1merror:[0m
       [31;1merror:[0m [35;1m[0mCannot build '[35;1m/nix/store/b7vx6aifdmz3a09rz89hkcaphagwq680-image-shell.json.drv[0m'.
       Reason: [31;1m1 dependency failed[0m.
       Output paths:
         [35;1m/nix/store/mswhq4nwznc6m65r204ga2qgx86lkn2p-image-shell.json[0m[0m
✖ Running command: /nix/store/3mjd0zanwzklk0bxn70cgn97m2dzd41w-devenv-nix-2.30.4/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option lazy-trees true --option warn-dirty false --keep-going --max-jobs 6 --option eval-cache false --option always-allow-substitutes true --option http-connections 100 --option extra-substituters https://devenv.cachix.org --option extra-trusted-public-keys  build --out-link /env/.devenv/gc/container-shell-derivation --print-out-paths -L .#devenv.perSystem.x86_64-linux.config.containers.shell.derivation in 145s
✖ Building shell container in 171s
Error:   × Command `/nix/store/3mjd0zanwzklk0bxn70cgn97m2dzd41w-devenv-nix-2.30.4/
  │ bin/nix --show-trace --extra-experimental-features nix-command --extra-
  │ experimental-features flakes --option lazy-trees true --option warn-dirty
  │ false --keep-going --max-jobs 6 --option eval-cache false --option always-
  │ allow-substitutes true --option http-connections 100 --option extra-
  │ substituters https://devenv.cachix.org --option extra-trusted-public-keys
  │ build --out-link /env/.devenv/gc/container-shell-derivation --print-out-
  │ paths -L .#devenv.perSystem.x86_64-
  │ linux.config.containers.shell.derivation -vv --log-format internal-json`
  │ failed with exit code 1
  • Full logs attached

logs.txt

Version

devenv 1.10.0 (x86_64-linux)

rob-mur avatar Nov 10 '25 11:11 rob-mur

I think I can confirm that this is a devenv issue because if I delete my lock file from within the container, run devenv update (getting back an identical lockfile in this case) then it works. I think it would be helpful if we had a devenv lock command to refresh the environments inputs as there must be something that happens when you add the input nix2container that isn't getting triggered in the container

rob-mur avatar Dec 02 '25 12:12 rob-mur

Found a workaround that at least successfully builds - no guarantees on if the produced container is correct yet. Essentially the issue is that devenv-container-tmp, root and etc are not copied to the container, but they are needed when building.

The below hack includes them exactly as written:

{
  pkgs,
  config,
  self,
  inputs,
  ...
}: let
  bash = "${pkgs.bashInteractive}/bin/bash";
  homeDir = "/env";
  user = "user";
  gid = "1000";
  group = "user";
  uid = "1000";

  mkRoot = pkgs.buildEnv {
    name = "devenv-container-root";
    paths = [
      pkgs.coreutils-full
      pkgs.bashInteractive
      pkgs.su
      pkgs.sudo
      pkgs.dockerTools.usrBinEnv
    ];
    pathsToLink = ["/bin" "/usr/bin"];
  };

  mkTmp = pkgs.runCommand "devenv-container-tmp" {} ''
    mkdir -p $out/tmp
  '';

  mkEtc = pkgs.runCommand "devenv-container-etc" {} ''
    mkdir -p $out/etc/pam.d

    echo "root:x:0:0:System administrator:/root:${bash}" > \
          $out/etc/passwd
    echo "${user}:x:${uid}:${gid}::${homeDir}:${bash}" >> \
          $out/etc/passwd

    echo "root:!x:::::::" > $out/etc/shadow
    echo "${user}:!x:::::::" >> $out/etc/shadow

    echo "root:x:0:" > $out/etc/group
    echo "${group}:x:${gid}:" >> $out/etc/group

    cat > $out/etc/pam.d/other <<EOF
    account sufficient pam_unix.so
    auth sufficient pam_rootok.so
    password requisite pam_unix.so nullok sha512
    session required pam_unix.so
    EOF

    touch $out/etc/login.defs
  '';
in {
  dotenv.enable = true;

  packages = with pkgs; [
    devenv
  ];

  containers.shell.layers = [
    {
      deps = [
        mkRoot
        mkTmp
        mkEtc
      ];
    }
  ];

  env = {
    LD_LIBRARY_PATH = "${pkgs.lib.makeLibraryPath config.packages}:${pkgs.stdenv.cc.cc.lib.outPath}/lib";
    SSL_CERT_FILE = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
    NIX_SSL_CERT_FILE = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
  };
}

rob-mur avatar Dec 03 '25 10:12 rob-mur

obviously this isn't maintainable so it would be better if src/modules/containers.nix was patched to do this automatically. I'll need to do some testing first that the produced containers are reliable but if so I can revisit this to fix it.

rob-mur avatar Dec 03 '25 10:12 rob-mur

See if https://github.com/cachix/devenv/pull/2334 makes it work :)

domenkozar avatar Dec 09 '25 12:12 domenkozar

Thanks, will take a look now. I've been using the containers following my patch approach seem to work fine, so assuming your fix works the built containers should be ok.

However, in the meantime I also noticed something you may want to include. Currently, rather than including the modified version of skopeo inside the container, it compiles it from source using go build on first use. However, that means that the first build inside the container is quite slow (adds about 7 minutes for me in a CI runner due to the CPU usage).

What you can do instead is include the modified skopeo inside the container and in that case it just uses it and things are performant as expected.

See my container patch below which applies both fixes.

devenv container patch 
{ pkgs
, inputs
, ...
}:
let
user = "user";
uid = "1000";
group = "user";
gid = "1000";
homeDir = "/env";
bash = "${pkgs.bashInteractive}/bin/bash";

mkRoot = pkgs.buildEnv {
  name = "devenv-container-root";
  paths = [
    pkgs.coreutils-full
    pkgs.bashInteractive
    pkgs.su
    pkgs.sudo
    pkgs.dockerTools.usrBinEnv
  ];
  pathsToLink = [ "/bin" "/usr/bin" ];
};

mkTmp = pkgs.runCommand "devenv-container-tmp" { } ''
  mkdir -p $out/tmp
'';
mkEtc = pkgs.runCommand "devenv-container-etc" { } ''
  mkdir -p $out/etc/pam.d

  echo "root:x:0:0:System administrator:/root:${bash}" > \
        $out/etc/passwd
  echo "${user}:x:${uid}:${gid}::${homeDir}:${bash}" >> \
        $out/etc/passwd

  echo "root:!x:::::::" > $out/etc/shadow
  echo "${user}:!x:::::::" >> $out/etc/shadow

  echo "root:x:0:" > $out/etc/group
  echo "${group}:x:${gid}:" >> $out/etc/group

  cat > $out/etc/pam.d/other <<EOF
  account sufficient pam_unix.so
  auth sufficient pam_rootok.so
  password requisite pam_unix.so nullok sha512
  session required pam_unix.so
  EOF

  touch $out/etc/login.defs
'';

nix2containerSkopeo = inputs.nix2container.packages.${pkgs.system};
in
{
containers.shell = {
  layers = [
    {
      deps = [
        mkRoot
        mkTmp
        mkEtc
        nix2containerSkopeo.nix2container-bin
        nix2containerSkopeo.skopeo-nix2container
      ];
    }
  ];
};
}

rob-mur avatar Jan 02 '26 12:01 rob-mur

Unfortunately I get an error with your branch. See my testbed.

The build succeeds but the copy then fails (this is the initial build, it doesn't succeed making any container):

Build output: /nix/store/s3w5m3spa1g71hx0yb82lvk6394j3w5j-stdenv-linux/setup: line 261: cd: vendor/github.com/containers/image/v5: No such file or directory
/nix/store/s3w5m3spa1g71hx0yb82lvk6394j3w5j-stdenv-linux/setup: line 261: cd: vendor/github.com/containers/image/v5: No such file or directory
Build completed
error: Cannot build '/nix/store/zndimx042wc6sdgkmfrkdar9lscp0k91-skopeo-1.21.0.drv'.
       Reason: builder failed with exit code 1.
       Output paths:
         /nix/store/1gip1b8s2rxfp699vj1ihrvycv3n7c98-skopeo-1.21.0
         /nix/store/4bq65fm1v4k4v1dpwpv575q4ncwdk27v-skopeo-1.21.0-man
       Last 8 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/c3jchrvy8v04hgw3wq9wffgkq4mhc398-source
       > source root is source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > Running phase: buildPhase
       > /nix/store/s3w5m3spa1g71hx0yb82lvk6394j3w5j-stdenv-linux/setup: line 261: cd: vendor/github.com/containers/image/v5: No such file or directory
       For full logs, run:
         nix log /nix/store/zndimx042wc6sdgkmfrkdar9lscp0k91-skopeo-1.21.0.drv
error: Cannot build '/nix/store/zndimx042wc6sdgkmfrkdar9lscp0k91-skopeo-1.21.0.drv'.
       Reason: builder failed with exit code 1.
       Output paths:
         /nix/store/1gip1b8s2rxfp699vj1ihrvycv3n7c98-skopeo-1.21.0
         /nix/store/4bq65fm1v4k4v1dpwpv575q4ncwdk27v-skopeo-1.21.0-man
       Last 8 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/c3jchrvy8v04hgw3wq9wffgkq4mhc398-source
       > source root is source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > Running phase: buildPhase
       > /nix/store/s3w5m3spa1g71hx0yb82lvk6394j3w5j-stdenv-linux/setup: line 261: cd: vendor/github.com/containers/image/v5: No such file or directory
       For full logs, run:
         nix log /nix/store/zndimx042wc6sdgkmfrkdar9lscp0k91-skopeo-1.21.0.drv
error:
       error: Cannot build '/nix/store/ygnqknn77slq2ly0bkbj1yqph78h2zx3-copy-container.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/cn7qzzcmdxpkvdnrkwhxr72dcxp00mq1-copy-container
error:
       error: Cannot build '/nix/store/ygnqknn77slq2ly0bkbj1yqph78h2zx3-copy-container.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/cn7qzzcmdxpkvdnrkwhxr72dcxp00mq1-copy-container
✖ Running command: /nix/store/407pr2lkcfqz8l79qdscw8lx95qa5hls-nix-devenv-2.30.0pre20251028_3e5644d/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option lazy-trees true --option warn-dirty false --keep-going --max-jobs 4 --optio
n eval-cache false --option always-allow-substitutes true --option http-connections 100 --option extra-substituters https://devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw= nixpkgs-python.cachix.org-1:hxjI7pFxTyuTHn
2NkvWCrAUcNZLNS3ZAvfYNuYifcEU= build --out-link /home/rob/Documents/Github/devenv/tests/container-nested-build/.devenv/gc/container-shell-copy --print-out-paths -L .#devenv.config.containers.shell.copyScript in 1.17s
✖ Evaluating Nix files in 1.16s
Error:   × Command `/nix/store/407pr2lkcfqz8l79qdscw8lx95qa5hls-nix-devenv-2.30.0pre20251028_3e5644d/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option lazy-trees true --option warn-dirty false --keep-going --max-jobs 4 --opti
on
  │ eval-cache false --option always-allow-substitutes true --option http-connections 100 --option extra-substituters https://devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw= nixpkgs-python.cachix.org-
  │ 1:hxjI7pFxTyuTHn2NkvWCrAUcNZLNS3ZAvfYNuYifcEU= build --out-link /home/rob/Documents/Github/devenv/tests/container-nested-build/.devenv/gc/container-shell-copy --print-out-paths -L .#devenv.config.containers.shell.copyScript -vv --log-format internal-json` failed with exit code
  │ 1

rob-mur avatar Jan 02 '26 12:01 rob-mur

This may be my setup so please let me know if so, otherwise feel free to play around with that testbed.

Also by the way, to be able to build containers within the container you will also need that cacert + SSL env setup that I've put in the testbed. Up to you if you want to include that by default or document it.

rob-mur avatar Jan 02 '26 12:01 rob-mur

You're unlucky and hitting https://github.com/nlewo/nix2container/pull/189

domenkozar avatar Jan 02 '26 18:01 domenkozar

Sorry for slow response.

So yes now with a more up to date devenv build I get passed that skopeo issue, however the container still doesnt build from within the container.

I've made a fresh repro as I needed to ensure the local devenv setup was getting included within the container.

Repro steps:

  • direnv allow to setup env
  • devenv container copy shell to make the container
  • docker run -it --entrypoint /bin/sh shell to launch it
  • cd into the repo
  • source $(./devenv.sh) to activate the shell
  • devenv container build -v shell

I've attached the error log that I got, but it ends with

error:                                                                                                                                                         
       error: Cannot build '/nix/store/fm6ycr8xpx6kwzrb3zbpgvqa8a0nd14i-layers.json.drv'.
       Reason: builder failed with exit code 1.
       Output paths:                                                           
         /nix/store/qzqlrbhscc99c9ly2dzlj8ggwcbmcj7w-layers.json
error:                                                                         
       error: Cannot build '/nix/store/fm6ycr8xpx6kwzrb3zbpgvqa8a0nd14i-layers.json.drv'.
       Reason: builder failed with exit code 1.
       Output paths:                                                           
         /nix/store/qzqlrbhscc99c9ly2dzlj8ggwcbmcj7w-layers.json
Build completed                                                                                                                                                
Build completed                                                                
Build completed                                                                
Build completed                                                                
error:                                                                         
       error: Cannot build '/nix/store/hy9gw5scynf01v3rdv45h9jq53h51y4b-image-shell.json.drv'.
       Reason: 1 dependency failed.                                            
       Output paths:                                                           
         /nix/store/m4zff1smfqcf38n0i9kv3wii98mbyxpm-image-shell.json
error:                                                                         
       error: Cannot build '/nix/store/hy9gw5scynf01v3rdv45h9jq53h51y4b-image-shell.json.drv'.
       Reason: 1 dependency failed.                                            
       Output paths:                                                           
         /nix/store/m4zff1smfqcf38n0i9kv3wii98mbyxpm-image-shell.json

rob-mur avatar Jan 18 '26 17:01 rob-mur

Log file was actually too large for gist or github comments directly, I've put it directly in the repro.

rob-mur avatar Jan 18 '26 17:01 rob-mur