Cannot put zfs in initrd: output is not allowed to refer to path ...-gcc-4.6.3

[ivan]

If I add "zfs" to boot.initrd.supportedFilesystems and run nixos-rebuild switch, I get a build failure ending in:

patching /nix/store/c3ayb2s6imrd2vn3xrmj4x58qzhb9l8r-extra-utils/bin/udevadm...
patching /nix/store/c3ayb2s6imrd2vn3xrmj4x58qzhb9l8r-extra-utils/bin/v4l_id...
testing patched programs...
hello world
BusyBox v1.20.2 (2013-05-23 19:40:56 UTC) multi-call binary.
203
Library version:   1.02.77 (2012-10-15)
  LVM version:     2.02.98(2) (2012-10-15)
mdadm - v3.1.2 - 10th March 2010
output is not allowed to refer to path `/nix/store/r7jz82h8d9fpqvyphcxr1hrqfnsp7q8v-gcc-4.6.3'
cannot build derivation `/nix/store/6b5ssw5wphqn93y2qh9lsmaycv57gy22-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/pl507q3giymxsiyab4wkz74f0xkifdqd-initrd.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/5c72l7brcdishlw8pw0znc7sx1q3055k-nixos-13.07pre4899_96be2d5-5405636.drv': 1 dependencies couldn't be built
error: build of `/nix/store/5c72l7brcdishlw8pw0znc7sx1q3055k-nixos-13.07pre4899_96be2d5-5405636.drv' failed

I am using the nixos-unstable channel, last updated today, 2013-07-29.

My /etc/nixos/configuration.nix:

{ config, pkgs, ... }:

{
  require =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  boot.initrd.kernelModules =
    [ # Specify all kernel modules that are necessary for mounting the root
      # filesystem.
      # "xfs" "ata_piix"
    ];

  boot.initrd.supportedFilesystems = [ "zfs" ];

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;

  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/sda";

  networking.hostName = "nixos-vanilla"; # Define your hostname.
  # networking.wireless.enable = true;  # Enables Wireless.

  # Add filesystem entries for each partition that you want to see
  # mounted at boot time.  This should include at least the root
  # filesystem.

  fileSystems."/".device = "/dev/disk/by-label/nixos-root";
  fileSystems."/boot".device = "/dev/disk/by-label/nixos-boot";

  # List swap partitions activated at boot time.
  swapDevices =
    [ # { device = "/dev/disk/by-label/swap"; }
    ];

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;
  services.openssh.permitRootLogin = "yes";

  services.xserver.enable = false;
}

I've pasted a full log of the rebuild to https://ludios.org/tmp/nixos-rebuild-zfs-in-initrd.txt because github is limited to 64KB of issue body text, and won't let me attach a text file.

[jcumming]

Huh. I suspect linking against libgcc_s to get pthread_cancel to for ztest might be the culprit.

Lemme check.

[jcumming]

Hrm. Rebuild worked. I'm trying to remember where I've seen that error before.

[ivan]

Seeing this now

'/nix/store/fh2ca1c7nhq5v08a01fxn12104km59np-e2fsprogs-1.42.7/lib/libext2fs.so.2.4' -> '/nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/lib/libext2fs.so.2.4'
'/nix/store/fh2ca1c7nhq5v08a01fxn12104km59np-e2fsprogs-1.42.7/lib/libss.so.2' -> '/nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/lib/libss.so.2'
'/nix/store/fh2ca1c7nhq5v08a01fxn12104km59np-e2fsprogs-1.42.7/lib/libss.so.2.0' -> '/nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/lib/libss.so.2.0'
stripping (with flags -s) in  /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/lib  /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin 
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/ata_id...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/blkid...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/busybox...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/cdrom_id...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/dmsetup...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/e2fsck...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/kmod...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/lvm...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/mdadm...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/scsi_id...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/systemd-udevd...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/tune2fs...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/udevadm...
patching /nix/store/jrw3wgbg2i2wg4smm3sv776czyccsl4z-extra-utils/bin/v4l_id...
testing patched programs...
hello world
BusyBox v1.20.2 (2013-05-23 19:40:56 UTC) multi-call binary.
206
Library version:   1.02.77 (2012-10-15)
  LVM version:     2.02.98(2) (2012-10-15)
mdadm - v3.1.2 - 10th March 2010
output is not allowed to refer to path `/nix/store/c93rqpqqn0w8xd0m0i9rv972mh7xqxxx-zfs-0.6.1-3.4.56'
cannot build derivation `/nix/store/xs717x6la7y58wwzsfmgz2a378067hy7-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/m6cgi80m6qp4wzlmgnci9qrgjlh5377i-initrd.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/a80fr2lcsqxbngn7z9488gkjjjdgjn02-nixos-13.07pre-9c0c41d-e2a8ff1.drv': 1 dependencies couldn't be built
error: build of `/nix/store/a80fr2lcsqxbngn7z9488gkjjjdgjn02-nixos-13.07pre-9c0c41d-e2a8ff1.drv' failed

maybe because I have zfs already installed this time

[jcumming]

I'm a bit out of my depth here, but could you try this patch:

diff --git a/modules/tasks/filesystems/zfs.nix b/modules/tasks/filesystems/zfs.nix
index 077d6a4..d1fdfd4 100644
--- a/modules/tasks/filesystems/zfs.nix
+++ b/modules/tasks/filesystems/zfs.nix
@@ -50,11 +50,15 @@ in

     boot.initrd = mkIf inInitrd { 
       kernelModules = [ "spl" "zfs" ] ;
+      # zfs uses libs from:  zfs, glibc, utillinux, zlib, gcc and glibc. 
+      # stage-1.nix provides:     glibc  utlilinux        gcc and glibc 
       extraUtilsCommands =
         ''
           cp -v ${kernel.zfs}/sbin/zfs $out/sbin
           cp -v ${kernel.zfs}/sbin/zdb $out/sbin
           cp -v ${kernel.zfs}/sbin/zpool $out/sbin
+          cp -pdv ${kernel.zfs}/lib/lib*.so.* $out/lib
+          cp -pdv ${pkgs.zlib}/lib/lib*.so.* $out/lib
         '';
       postDeviceCommands =

[ivan]

Same thing, I think

[at@nixos:~/sources/nixos]$ git diff
diff --git a/modules/tasks/filesystems/zfs.nix b/modules/tasks/filesystems/zfs.nix
index 590eb00..968748a 100644
--- a/modules/tasks/filesystems/zfs.nix
+++ b/modules/tasks/filesystems/zfs.nix
@@ -50,11 +50,15 @@ in

     boot.initrd = mkIf inInitrd { 
       kernelModules = [ "spl" "zfs" ] ;
+      # zfs uses libs from:  zfs, glibc, utillinux, zlib, gcc and glibc. 
+      # stage-1.nix provides:     glibc  utlilinux        gcc and glibc 
       extraUtilsCommands =
         ''
           cp -v ${kernel.zfs}/sbin/zfs $out/sbin
           cp -v ${kernel.zfs}/sbin/zdb $out/sbin
           cp -v ${kernel.zfs}/sbin/zpool $out/sbin
+          cp -pdv ${kernel.zfs}/lib/lib*.so.* $out/lib
+          cp -pdv ${pkgs.zlib}/lib/lib*.so.* $out/lib
         '';
       postDeviceCommands =
         ''

[at@nixos:~/sources/nixos]$ 

# nixos-rebuild -I /home/at/sources switch
[...]
'/nix/store/fh2ca1c7nhq5v08a01fxn12104km59np-e2fsprogs-1.42.7/lib/libss.so.2.0' -> '/nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/lib/libss.so.2.0'
stripping (with flags -s) in  /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/lib  /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin 
building /nix/store/hr48wlvcsdysr420bm0dzl30cwyb8br3-unit
building path(s) `/nix/store/nfnapb6a8i2dg8dw5ilv0bpany1q1wzv-units'
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/ata_id...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/blkid...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/busybox...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/cdrom_id...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/dmsetup...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/e2fsck...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/kmod...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/lvm...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/mdadm...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/scsi_id...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/systemd-udevd...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/tune2fs...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/udevadm...
patching /nix/store/sqin4n8ys8790a9wjcpi1j4rr41dcw1h-extra-utils/bin/v4l_id...
testing patched programs...
hello world
BusyBox v1.20.2 (2013-05-23 19:40:56 UTC) multi-call binary.
206
Library version:   1.02.77 (2012-10-15)
  LVM version:     2.02.98(2) (2012-10-15)
mdadm - v3.1.2 - 10th March 2010
output is not allowed to refer to path `/nix/store/c93rqpqqn0w8xd0m0i9rv972mh7xqxxx-zfs-0.6.1-3.4.56'
cannot build derivation `/nix/store/rp2fk783p9n8b8ic39dm86ggpj8aw5kw-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/xqjv5k4apv66b4ij089ww0q0svzr3n5p-initrd.drv': 1 dependencies couldn't be built
cannot build derivation `/nix/store/5c4axywnn2zyw66v9dl9dmgm1ljgf97h-nixos-13.07pre-9c0c41dM-e2a8ff1.drv': 1 dependencies couldn't be built
error: build of `/nix/store/5c4axywnn2zyw66v9dl9dmgm1ljgf97h-nixos-13.07pre-9c0c41dM-e2a8ff1.drv' failed

[ivan]

jcumming discovered the culprit (patchelf not being run on sbin/*). My initrd builds successfully with this patch, though jcumming is considering making zfs install to bin/ instead.

diff --git a/modules/system/boot/stage-1.nix b/modules/system/boot/stage-1.nix
index 9efae22..81a82ca 100644
--- a/modules/system/boot/stage-1.nix
+++ b/modules/system/boot/stage-1.nix
@@ -162,7 +162,7 @@ let
       doublePatchelf = pkgs.stdenv.isArm;
     }
     ''
-      mkdir -p $out/bin $out/lib
+      mkdir -p $out/bin $out/sbin $out/lib

       # Copy what we need from Glibc.
       cp -pv ${pkgs.glibc}/lib/ld*.so.? $out/lib
@@ -217,12 +217,12 @@ let

       # Strip binaries further than normal.
       chmod -R u+w $out
-      stripDirs "lib bin" "-s"
+      stripDirs "lib bin sbin" "-s"

       # Run patchelf to make the programs refer to the copied libraries.
-      for i in $out/bin/* $out/lib/*; do if ! test -L $i; then nuke-refs $i; fi; done
+      for i in $out/bin/* $out/sbin/* $out/lib/*; do if ! test -L $i; then nuke-refs $i; fi; done

-      for i in $out/bin/*; do
+      for i in $out/bin/* $out/sbin/*; do
           if ! test -L $i; then
               echo "patching $i..."
               patchelf --set-interpreter $out/lib/ld*.so.? --set-rpath $out/lib $i || true

[edolstra]

Why not just copy everything to $out/bin?

[alexanderkjeldaas]

FYI, I was hit by this issue as well. Completely unrelated issue originally, but I put stuff in $out/sbin by mistake.