try icon indicating copy to clipboard operation
try copied to clipboard

Fails on NixOS w/ btrfs

Open GabrielDTB opened this issue 2 years ago • 1 comments

Running NixOS with brtfs and the following disk layout:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   2.7T  0 disk  
└─sda1        8:1    0   2.7T  0 part  
sdb           8:16   0   2.7T  0 disk  
└─sdb1        8:17   0   2.7T  0 part  
sdc           8:32   0 476.9G  0 disk  
└─sdc1        8:33   0 476.9G  0 part  
sdd           8:48   1     7M  0 disk  
└─sdd1        8:49   1     7M  0 part  
nvme0n1     259:0    0 238.5G  0 disk  
├─nvme0n1p1 259:1    0   512M  0 part  /boot
├─nvme0n1p2 259:2    0   192G  0 part  /home
│                                      /nix/store
│                                      /
└─nvme0n1p3 259:3    0    46G  0 part  
  └─enclave 254:0    0    46G  0 crypt /swap

Executing ./try echo hi on mount-fix branch errors out with warnings that every mount point failed to mount as an overlay. It appears that this is caused by failed uid/gid mappings that must occur due to NixOS's wrapper.c. Full command output and contents of the referenced tmp file are included below.

$ ./try echo hi
/tmp/tmp.AOXiqDxls5: line 4:     3 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /bin as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:     7 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /etc as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    10 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /mnt as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    12 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /nix as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    15 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /root as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    18 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /srv as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    22 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /tmp as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    24 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /usr as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 4:    26 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir=/"$top_dir",upperdir="$SANDBOX_DIR"/upperdir/"$top_dir",workdir="$SANDBOX_DIR"/workdir/"$top_dir" "$SANDBOX_DIR"/temproot/"$top_dir" 2>> "$try_mount_log"
Warning: Failed mounting /var as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 20:    30 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir="$mount_dir",upperdir="$SANDBOX_DIR"/upperdir"$mount_dir",workdir="$SANDBOX_DIR"/workdir"$mount_dir" "$SANDBOX_DIR"/temproot"$mount_dir" 2>> "$try_mount_log"
Warning: Failed mounting /run/user/1000/doc as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 20:    31 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir="$mount_dir",upperdir="$SANDBOX_DIR"/upperdir"$mount_dir",workdir="$SANDBOX_DIR"/workdir"$mount_dir" "$SANDBOX_DIR"/temproot"$mount_dir" 2>> "$try_mount_log"
Warning: Failed mounting /nix/store as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 20:    32 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir="$mount_dir",upperdir="$SANDBOX_DIR"/upperdir"$mount_dir",workdir="$SANDBOX_DIR"/workdir"$mount_dir" "$SANDBOX_DIR"/temproot"$mount_dir" 2>> "$try_mount_log"
Warning: Failed mounting /swap as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 20:    33 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir="$mount_dir",upperdir="$SANDBOX_DIR"/upperdir"$mount_dir",workdir="$SANDBOX_DIR"/workdir"$mount_dir" "$SANDBOX_DIR"/temproot"$mount_dir" 2>> "$try_mount_log"
Warning: Failed mounting /boot as an overlay, see /tmp/tmp.dh9r3FP2pr
/tmp/tmp.AOXiqDxls5: line 20:    34 Aborted                 (core dumped) mount -t overlay overlay -o lowerdir="$mount_dir",upperdir="$SANDBOX_DIR"/upperdir"$mount_dir",workdir="$SANDBOX_DIR"/workdir"$mount_dir" "$SANDBOX_DIR"/temproot"$mount_dir" 2>> "$try_mount_log"
Warning: Failed mounting /home as an overlay, see /tmp/tmp.dh9r3FP2pr
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
/tmp/tmp.AOXiqDxls5: line 28:    35 Aborted                 (core dumped) mount --rbind /dev "$SANDBOX_DIR/temproot/dev"
/tmp/tmp.AOXiqDxls5: line 31:    36 Aborted                 (core dumped) mount --rbind --read-only /run "$SANDBOX_DIR/temproot/run" 2> /dev/null
unshare: failed to execute /bin/bash: No such file or directory
$ cat tmp.dh9r3FP2pr 
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.
Assertion `!(st.st_mode & S_ISUID) || (st.st_uid == geteuid())` in NixOS's wrapper.c failed.

Running the same command on future branch, after running setup.sh as root, gives:

$ ./try echo hi
nc: Address already in use
./try: line 225: ./gidmapper: cannot execute: required file not found
nc: invalid option -- 'q'
usage: nc [-46cDdFhklNnrStUuvz] [-C certfile] [-e name] [-H hash] [-I length]
	 [-i interval] [-K keyfile] [-M ttl] [-m minttl] [-O length]
	 [-o staplefile] [-P proxy_username] [-p source_port] [-R CAfile]
	 [-s sourceaddr] [-T keyword] [-V rtable] [-W recvlimit] [-w timeout]
	 [-X proxy_protocol] [-x proxy_address[:port]] [-Z peercertfile]
	 [destination] [port]
^C

GabrielDTB avatar Jun 25 '23 00:06 GabrielDTB

@ericzty

GabrielDTB avatar Jun 25 '23 00:06 GabrielDTB

The (wrapped) one in Nixpkgs works fine, just needed to install mergerfs as well

ErrorNoInternet avatar Dec 13 '23 16:12 ErrorNoInternet

The (wrapped) one in Nixpkgs works fine, just needed to install mergerfs as well

Thank you for the update, are you using btrfs aswell or ext4 or some other fs?

ezrizhu avatar Dec 13 '23 18:12 ezrizhu

Thank you for the update, are you using btrfs aswell or ext4 or some other fs?

btrfs as well :)

ErrorNoInternet avatar Dec 13 '23 18:12 ErrorNoInternet