Test derivation can't be build, bash missing

[sheepforce]

I am trying to run nix-portable on our CentOS HPC cluster (CentOS Linux release 7.6.1810 (Core), Linux login01 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux). The home directories are on a NFS and namespaces are not available.

When trying to run nix-portable I get:

 NP_DEBUG=1 nix-portable nix-shell -p nano
installing files
figuring out ssl certs
SSL_CERT_FILE not defined. trying to find certs automatically
found /etc/ssl/certs/ca-bundle.crt with real path /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
figuring out which runtime to use
bwrap executable: /home/mi24ris/.nix-portable/bin/bwrap
proot executable: /home/mi24ris/.nix-portable/bin/proot
bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces.  bwrap must be installed setuid on such systems.
bwrap doesn't work on this system -> will use proot
base command will be: /home/mi24ris/.nix-portable/bin/proot -r /home/mi24ris/.nix-portable/emptyroot -b /dev:/dev -b /home/mi24ris/.nix-portable/store:/nix/store -b /proc:/proc -b /run:/run -b /sys:/sys -b /tmp:/tmp -b /etc:/etc -b /root:/root -b /var:/var -b /usr:/usr -b /usr/bin:/bin -b /usr/sbin:/sbin -b /usr/lib:/lib -b /usr/lib64:/lib64 -b /boot:/boot -b /home:/home -b /media:/media -b /mnt:/mnt -b /opt:/opt -b /srv:/srv -b /.autorelabel:/.autorelabel -b /cluster:/cluster -b /beegfs:/beegfs -b /.readahead:/.readahead -b /backup1:/backup1 -b /backup2:/backup2 -b /xcatpost:/xcatpost -b /applications-merged:/applications-merged -b /local:/local -b /tpi1:/tpi1 -b /nfsdata:/nfsdata -b /hs_err_pid26423.log:/hs_err_pid26423.log -b /hs_err_pid26386.log:/hs_err_pid26386.log -b /hs_err_pid35596.log:/hs_err_pid35596.log -b /hs_err_pid30100.log:/hs_err_pid30100.log -b /hs_err_pid45475.log:/hs_err_pid45475.log -b /hs_err_pid45597.log:/hs_err_pid45597.log -b /hs_err_pid45631.log:/hs_err_pid45631.log -b /-:/- -b /hs_err_pid33509.log:/hs_err_pid33509.log -b /hs_err_pid34531.log:/hs_err_pid34531.log -b /hs_err_pid34650.log:/hs_err_pid34650.log -b /hs_err_pid35790.log:/hs_err_pid35790.log -b /hs_err_pid35835.log:/hs_err_pid35835.log -b /nfs04:/nfs04 -b /hs_err_pid38932.log:/hs_err_pid38932.log -b /etc/host.conf:/etc/host.conf -b /etc/hosts:/etc/hosts -b /etc/networks:/etc/networks -b /etc/passwd:/etc/passwd -b /etc/group:/etc/group -b /etc/nsswitch.conf:/etc/nsswitch.conf -b /etc/resolv.conf:/etc/resolv.conf -b /usr/share/zoneinfo/Europe/Berlin:/usr/share/zoneinfo/Europe/Berlin -b /home/mi24ris:/home/mi24ris -b /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/home/mi24ris/.nix-portable/ca-bundle.crt
Testing if nix can build stuff without sandbox
this derivation will be built:
  /nix/store/d1dixwac2sq99s6zxss75xv7nzy0qk1x-test.drv
building '/nix/store/d1dixwac2sq99s6zxss75xv7nzy0qk1x-test.drv'...

ErroSysError~executing '/nix/store/9ywr69qi622lrmx5nn88gk8jpmihy0dz-bash-4.4-p23/bin/bash': No such file or directory
error: builder for '/nix/store/d1dixwac2sq99s6zxss75xv7nzy0qk1x-test.drv' failed with exit code 1
Fatal error: nix is unable to build packages

And the store is also actually missing this bash. I am a little bit clueless what's happening, as it works for another user on the cluster.

[DavHau]

Thanks for the report. Is there any noticable difference between using nix-portable v008 vs. v009?

[sheepforce]

Thanks for looking into it :) There is some difference in the messages, yes. But both don't work. Here is the output for v008.

installing files
figuring out ssl certs
SSL_CERT_FILE not defined. trying to find certs automatically
found /etc/ssl/certs/ca-bundle.crt with real path /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
figuring out which runtime to use
bwrap executable: /home/mi24ris/.nix-portable/bin/bwrap
proot executable: /home/mi24ris/.nix-portable/bin/proot
bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces.  bwrap must be installed setuid on such systems.
bwrap doesn't work on this system -> will use proot
base command will be: /home/mi24ris/.nix-portable/bin/proot -r /home/mi24ris/.nix-portable/emptyroot -b /dev:/dev -b /home/mi24ris/.nix-portable/store:/nix/store -b /proc:/proc -b /run:/run -b /sys:/sys -b /tmp:/tmp -b /etc:/etc -b /root:/root -b /var:/var -b /usr:/usr -b /usr/bin:/bin -b /usr/sbin:/sbin -b /usr/lib:/lib -b /usr/lib64:/lib64 -b /boot:/boot -b /home:/home -b /media:/media -b /mnt:/mnt -b /opt:/opt -b /srv:/srv -b /.autorelabel:/.autorelabel -b /cluster:/cluster -b /beegfs:/beegfs -b /.readahead:/.readahead -b /backup1:/backup1 -b /backup2:/backup2 -b /xcatpost:/xcatpost -b /applications-merged:/applications-merged -b /local:/local -b /tpi1:/tpi1 -b /nfsdata:/nfsdata -b /hs_err_pid26423.log:/hs_err_pid26423.log -b /hs_err_pid26386.log:/hs_err_pid26386.log -b /hs_err_pid35596.log:/hs_err_pid35596.log -b /hs_err_pid30100.log:/hs_err_pid30100.log -b /hs_err_pid45475.log:/hs_err_pid45475.log -b /hs_err_pid45597.log:/hs_err_pid45597.log -b /hs_err_pid45631.log:/hs_err_pid45631.log -b /-:/- -b /hs_err_pid33509.log:/hs_err_pid33509.log -b /hs_err_pid34531.log:/hs_err_pid34531.log -b /hs_err_pid34650.log:/hs_err_pid34650.log -b /hs_err_pid35790.log:/hs_err_pid35790.log -b /hs_err_pid35835.log:/hs_err_pid35835.log -b /nfs04:/nfs04 -b /hs_err_pid38932.log:/hs_err_pid38932.log -b /etc/host.conf:/etc/host.conf -b /etc/hosts:/etc/hosts -b /etc/networks:/etc/networks -b /etc/passwd:/etc/passwd -b /etc/group:/etc/group -b /etc/nsswitch.conf:/etc/nsswitch.conf -b /etc/resolv.conf:/etc/resolv.conf -b /usr/share/zoneinfo/Europe/Berlin:/usr/share/zoneinfo/Europe/Berlin -b /home/mi24ris:/home/mi24ris -b /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/home/mi24ris/.nix-portable/ca-bundle.crt
cat: can't open '/home/mi24ris/.nix-portable/conf/last_runtime': No such file or directory
Testing if nix can build stuff without sandbox
proot error: execve("/nix/store/p8d4qqiqcmx935m2b5a1gsmr6sp1ihsn-nix-2.4pre20201201_5a6ddb3/bin/nix-build"): No such file or directory
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot error: trying to remove a directory outside of '/tmp', please report this error.

proot error: can't chmod '/tmp/proot-4711-uZ5uEW': No such file or directory
Fatal error: nix is unable to build packages

[DavHau]

For the store path which it complains not to be found, could you check if it exists in /home/mi24ris/.nix-portable/store. So for your last try with v008 the path to check would be: /home/mi24ris/.nix-portable/store/p8d4qqiqcmx935m2b5a1gsmr6sp1ihsn-nix-2.4pre20201201_5a6ddb3/bin/nix-build Does it exist and is the file executable for your user?

Otherwise something went wrong already during nix-store-archive extraction. Maybe in that case you could try deleting the whole $HOME/.nix-portable directory and see if a fresh run fixes the problem.

If the file is there and the permissions are alright, then the problem must be with proot in conjunction with your system. In that case you could try to build nix-portable against an other/older version of proot. But that all is just poking around in the dark and you might waste your time.

Also you could try experimenting with proot alone and see if a simple bind mount of some directory into your $HOMR works or not.

[sheepforce]

OK, completely clean with chmod u+rwx -R ~/.nix* && rm -rf ~/.nix* results for the v009 version. NP_DEBUG=1 nix-portable nix-shell -p nano fails differently this time with ErroSysErrorpexecuting '/nix/store/lan2w3ab1mvpxj3ppiw2sizh8i7rpz7s-busybox': No such file or directory and this file is also actually missing. However, proot seems to work and I can enter the environment with the command NP_DEBUG gives:

$ /home/mi24ris/.nix-portable/bin/proot -r /home/mi24ris/.nix-portable/emptyroot -b /dev:/dev -b /home/mi24ris/.nix-portable/store:/nix/store -b /proc:/proc -b /run:/run -b /sys:/sys -b /tmp:/tmp -b /etc:/etc -b /root:/root -b /var:/var -b /usr:/usr -b /usr/bin:/bin -b /usr/sbin:/sbin -b /usr/lib:/lib -b /usr/lib64:/lib64 -b /boot:/boot -b /home:/home -b /media:/media -b /mnt:/mnt -b /opt:/opt -b /srv:/srv -b /.autorelabel:/.autorelabel -b /cluster:/cluster -b /beegfs:/beegfs -b /.readahead:/.readahead -b /backup1:/backup1 -b /backup2:/backup2 -b /xcatpost:/xcatpost -b /applications-merged:/applications-merged -b /local:/local -b /tpi1:/tpi1 -b /nfsdata:/nfsdata -b /hs_err_pid26423.log:/hs_err_pid26423.log -b /hs_err_pid26386.log:/hs_err_pid26386.log -b /hs_err_pid35596.log:/hs_err_pid35596.log -b /hs_err_pid30100.log:/hs_err_pid30100.log -b /hs_err_pid45475.log:/hs_err_pid45475.log -b /hs_err_pid45597.log:/hs_err_pid45597.log -b /hs_err_pid45631.log:/hs_err_pid45631.log -b /-:/- -b /hs_err_pid33509.log:/hs_err_pid33509.log -b /hs_err_pid34531.log:/hs_err_pid34531.log -b /hs_err_pid34650.log:/hs_err_pid34650.log -b /hs_err_pid35790.log:/hs_err_pid35790.log -b /hs_err_pid35835.log:/hs_err_pid35835.log -b /nfs04:/nfs04 -b /hs_err_pid38932.log:/hs_err_pid38932.log -b /etc/host.conf:/etc/host.conf -b /etc/hosts:/etc/hosts -b /etc/networks:/etc/networks -b /etc/passwd:/etc/passwd -b /etc/group:/etc/group -b /etc/nsswitch.conf:/etc/nsswitch.conf -b /etc/resolv.conf:/etc/resolv.conf -b /usr/share/zoneinfo/Europe/Berlin:/usr/share/zoneinfo/Europe/Berlin -b /home/mi24ris:/home/mi24ris -b /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/home/mi24ris/.nix-portable/ca-bundle.crt
$ ls -lah /nix/store/ | grep busybox
drwxr-xr-x  3 mi24ris uj08g-theochem   51 Feb  1 16:58 q4ihv2kv8z9nzwpsjkinig81p7mbayg4-busybox-static-x86_64-unknown-linux-musl-1.34.1
drwxr-xr-x  3 mi24ris uj08g-theochem   51 Feb  1 10:57 vghpblp2s4lvzqgczmgprmzvid9xs7md-busybox-1.31.1-x86_64-unknown-linux-musl

[DavHau]

I have problems understanding why exactly it tries to access that busybox path. This store path doesn't exist as well on my test machine. Could you run this with NP_DEBUG=2 and see at which command it fails exactly?

[sheepforce]

Sure, here is the log with NP_DEBUG=2 NP_DEBUG.log

[tennox]

I encountered a similar problem and fixed it by sudo rm -rf ~/.nix-portable :stuck_out_tongue_closed_eyes: