NixOS
Nix daemon blocked by sandbox on macOS Monterey (on external SSD)
Describe the bug
After installing nix the nix daemon won't run since it's being blocked by sandboxing from opening files in the nix store. This is happening on macOS Monterey installed on and booted from an external SSD.
Steps To Reproduce
- (Install and boot MacOS on an external SSD.)
- Follow the installation instructions for Nix on MacOS:
sh <(curl -L https://nixos.org/nix/install) - Complete the installation steps. (Notice that the Nix store volume was created on the external SSD.)
- Check if nix is running:
ps -ef | grep nix. It's not. (Occasionally a wait4path is running if darwin-store daemon hasn't started correctly.) - Check nix-daemon log:
tail /private/var/log/nix-daemon.log
dyld[1397]: Library not loaded: /nix/store/95n0y1033piss1slf99is61a3zc0yv0p-libsodium-1.0.18/lib/libsodium.23.dylib
Referenced from: /nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemon
Reason: tried: '/nix/store/95n0y1033piss1slf99is61a3zc0yv0p-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file)
(Repeated over and over again.)
So libsodium can't be loaded due to sandbox rules.
Expected behavior
nix-deamon and darwin-store running
nix-env --version output
Additional context
Note that I'm running MacOS on a FileVault encrypted external SSD (with the internal disk locked). I don't know for sure that this is related to the problem, but as it's probably not the most usual setup it might be the reason why I encounter this issue while others can install Nix on MacOS without problem.
I haven't run it on an external disk, but I would imagine someone has. I don't know what this would be off the top of my head, but I guess we can start picking at it. A few Qs:
- What do these diskutil commands report?
diskutil listdiskutil apfs listdiskutil info /diskutil info /nix
- What happens if you invoke
/nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemondirectly from a terminal? - What does
with the internal disk lockedmean? The disk is encrypted and the credential isn't in keychain?
Hi, thanks for helping out.
-
Running
sudo /nix/store/hyfj5imsd0c4amlcjpf8l6w4q2draaj3-nix-2.7.0/bin/nix-daemonworked. The deamon is running and I can use e.g.nix-shell -p hello. Probably should have checked this ... But I still don't know why it's different when it's started as a launch deamon. -
It means exactly that. Encrypted with FileVault and the credentials not in the keychain.
.
- Here is the output of each of the commands:
MacBook-Pro% diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *500.3 GB disk0
1: EFI EFI 314.6 MB disk0s1
2: Apple_APFS Container disk1 500.0 GB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +500.0 GB disk1
Physical Store disk0s2
1: APFS Volume Macintosh HD – data 342.6 GB disk1s1
2: APFS Volume Preboot 342.0 MB disk1s2
3: APFS Volume Recovery 626.5 MB disk1s3
4: APFS Volume VM 6.4 GB disk1s4
5: APFS Volume Macintosh HD 15.3 GB disk1s5
6: APFS Volume Nix Store 29.7 GB disk1s7
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *500.1 GB disk2
1: EFI EFI 209.7 MB disk2s1
2: Apple_APFS Container disk3 499.9 GB disk2s2
/dev/disk3 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +499.9 GB disk3
Physical Store disk2s2
1: APFS Volume Komprimatorn SSD – data 368.7 GB disk3s1
2: APFS Volume Preboot 271.2 MB disk3s2
3: APFS Volume Recovery 1.1 GB disk3s3
4: APFS Volume VM 8.6 GB disk3s4
5: APFS Volume Komprimatorn SSD 15.7 GB disk3s5
6: APFS Snapshot com.apple.os.update-... 15.7 GB disk3s5s1
7: APFS Volume Nix Store 494.5 MB disk3s7
MacBook-Pro% diskutil apfs list
APFS Containers (2 found)
|
+-- Container disk1 30815887-19CF-4A17-8E8E-169D1741EFB1
| ====================================================
| APFS Container Reference: disk1
| Size (Capacity Ceiling): 499963170816 B (500.0 GB)
| Capacity In Use By Volumes: 395193638912 B (395.2 GB) (79.0% used)
| Capacity Not Allocated: 104769531904 B (104.8 GB) (21.0% free)
| |
| +-< Physical Store disk0s2 3AFBE9E5-6BEE-4626-9865-11BB102F293E
| | -----------------------------------------------------------
| | APFS Physical Store Disk: disk0s2
| | Size: 499963170816 B (500.0 GB)
| |
| +-> Volume disk1s1 591DEFF8-8B2F-35D1-827A-2ED22C268720
| | ---------------------------------------------------
| | APFS Volume Disk (Role): disk1s1 (Data)
| | Name: Macintosh HD – data (Case-insensitive)
| | Mount Point: Not Mounted
| | Capacity Consumed: 342629449728 B (342.6 GB)
| | Sealed: No
| | FileVault: Yes (Locked)
| |
| +-> Volume disk1s2 20AE12A9-A39E-476C-9361-DC34DC53021D
| | ---------------------------------------------------
| | APFS Volume Disk (Role): disk1s2 (Preboot)
| | Name: Preboot (Case-insensitive)
| | Mount Point: Not Mounted
| | Capacity Consumed: 341970944 B (342.0 MB)
| | Sealed: No
| | FileVault: No
| |
| +-> Volume disk1s3 1938929C-1561-4F29-9BE7-EE93237D65CD
| | ---------------------------------------------------
| | APFS Volume Disk (Role): disk1s3 (Recovery)
| | Name: Recovery (Case-insensitive)
| | Mount Point: Not Mounted
| | Capacity Consumed: 626454528 B (626.5 MB)
| | Sealed: No
| | FileVault: No
| |
| +-> Volume disk1s4 6FCCF702-7C1C-46A8-AACD-4449F226C3C5
| | ---------------------------------------------------
| | APFS Volume Disk (Role): disk1s4 (VM)
| | Name: VM (Case-insensitive)
| | Mount Point: Not Mounted
| | Capacity Consumed: 6443610112 B (6.4 GB)
| | Sealed: No
| | FileVault: No
| |
| +-> Volume disk1s5 33E7EE67-CED5-4A70-9AAC-FF52BA1F13AB
| | ---------------------------------------------------
| | APFS Volume Disk (Role): disk1s5 (System)
| | Name: Macintosh HD (Case-insensitive)
| | Mount Point: /Volumes/Macintosh HD
| | Capacity Consumed: 15321980928 B (15.3 GB)
| | Sealed: Yes
| | FileVault: Yes (Locked)
| | Encrypted: No
| |
| +-> Volume disk1s7 D0A98DB1-1115-4CEA-BE4D-86CED0EFBAAA
| ---------------------------------------------------
| APFS Volume Disk (Role): disk1s7 (No specific role)
| Name: Nix Store (Case-insensitive)
| Mount Point: Not Mounted
| Capacity Consumed: 29667000320 B (29.7 GB)
| Sealed: No
| FileVault: Yes (Locked)
|
+-- Container disk3 5B1B8E0D-EB85-4019-A4D7-89AC7DDFA889
====================================================
APFS Container Reference: disk3
Size (Capacity Ceiling): 499898105856 B (499.9 GB)
Capacity In Use By Volumes: 395098595328 B (395.1 GB) (79.0% used)
Capacity Not Allocated: 104799510528 B (104.8 GB) (21.0% free)
|
+-< Physical Store disk2s2 9F148CE8-A8D5-4747-9642-39FDFAE5A673
| -----------------------------------------------------------
| APFS Physical Store Disk: disk2s2
| Size: 499898105856 B (499.9 GB)
|
+-> Volume disk3s1 C138F899-4812-31CE-AADB-AD396FCBBA79
| ---------------------------------------------------
| APFS Volume Disk (Role): disk3s1 (Data)
| Name: Komprimatorn SSD – data (Case-insensitive)
| Mount Point: /System/Volumes/Data
| Capacity Consumed: 368734658560 B (368.7 GB)
| Sealed: No
| FileVault: Yes (Unlocked)
|
+-> Volume disk3s2 59691721-815D-42C8-9D1E-682FE212115C
| ---------------------------------------------------
| APFS Volume Disk (Role): disk3s2 (Preboot)
| Name: Preboot (Case-insensitive)
| Mount Point: /System/Volumes/Preboot
| Capacity Consumed: 271163392 B (271.2 MB)
| Sealed: No
| FileVault: No
|
+-> Volume disk3s3 AAF19554-2813-43CD-9A81-8F9464F7F4A4
| ---------------------------------------------------
| APFS Volume Disk (Role): disk3s3 (Recovery)
| Name: Recovery (Case-insensitive)
| Mount Point: Not Mounted
| Capacity Consumed: 1095483392 B (1.1 GB)
| Sealed: No
| FileVault: No
|
+-> Volume disk3s4 604F1841-C79B-40D7-94CB-1DFFCF7FAEC5
| ---------------------------------------------------
| APFS Volume Disk (Role): disk3s4 (VM)
| Name: VM (Case-insensitive)
| Mount Point: /System/Volumes/VM
| Capacity Consumed: 8591003648 B (8.6 GB)
| Sealed: No
| FileVault: No
|
+-> Volume disk3s5 FF642800-AB81-43D1-A9E2-2F55C97A2656
| ---------------------------------------------------
| APFS Volume Disk (Role): disk3s5 (System)
| Name: Komprimatorn SSD (Case-insensitive)
| Mount Point: Not Mounted
| Capacity Consumed: 15749931008 B (15.7 GB)
| Sealed: Yes
| FileVault: Yes (Unlocked)
| Encrypted: No
| |
| Snapshot: 7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
| Snapshot Disk: disk3s5s1
| Snapshot Mount Point: /
| Snapshot Sealed: Yes
|
+-> Volume disk3s7 FF6E9075-BB92-4631-81DA-3481B13C2470
---------------------------------------------------
APFS Volume Disk (Role): disk3s7 (No specific role)
Name: Nix Store (Case-insensitive)
Mount Point: /nix
Capacity Consumed: 494452736 B (494.5 MB)
Sealed: No
FileVault: Yes (Unlocked)
MacBook-Pro% diskutil info /
Device Identifier: disk3s5s1
Device Node: /dev/disk3s5s1
Whole: No
Part of Whole: disk3
Volume Name: Komprimatorn SSD
Mounted: Yes
Mount Point: /
Partition Type: 41504653-0000-11AA-AA11-00306543ECAC
File System Personality: APFS
Type (Bundle): apfs
Name (User Visible): APFS
Owners: Enabled
OS Can Be Installed: No
Booter Disk: disk3s2
Recovery Disk: disk3s3
Media Type: Generic
Protocol: USB
SMART Status: Not Supported
Volume UUID: 7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
Disk / Partition UUID: 7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
Disk Size: 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
Device Block Size: 4096 Bytes
Container Total Space: 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
Container Free Space: 104.8 GB (104795762688 Bytes) (exactly 204679224 512-Byte-Units)
Allocation Block Size: 4096 Bytes
Media OS Use Only: No
Media Read-Only: Yes
Volume Read-Only: Yes (read-only mount flag set)
Device Location: External
Removable Media: Fixed
Solid State: Yes
This disk is an APFS Volume Snapshot. APFS Information:
APFS Snapshot Name: com.apple.os.update-9D831579E656FF2E36BAC3334521407031CBA216B7079C5998A817A5EC7C76B7
APFS Snapshot UUID: 7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
APFS Container: disk3
APFS Physical Store: disk2s2
Fusion Drive: No
APFS Volume Group: C138F899-4812-31CE-AADB-AD396FCBBA79
EFI Driver In macOS: 1933080003000000
Encrypted: No
FileVault: Yes
Sealed: Yes
Locked: No
APFS Snapshots are defined upon this APFS Volume. Snapshot list:
Snapshot UUID: 7A5E5ADC-C3EA-4C5E-9AE1-5AD8499F08CD
Name: com.apple.os.update-9D831579E656FF2E36BAC3334521407031CBA216B7079C5998A817A5EC7C76B7
XID: 50989
MacBook-Pro% diskutil info /nix
Device Identifier: disk3s7
Device Node: /dev/disk3s7
Whole: No
Part of Whole: disk3
Volume Name: Nix Store
Mounted: Yes
Mount Point: /nix
Partition Type: 41504653-0000-11AA-AA11-00306543ECAC
File System Personality: APFS
Type (Bundle): apfs
Name (User Visible): APFS
Owners: Enabled
OS Can Be Installed: Yes
Booter Disk: disk3s2
Recovery Disk: disk3s3
Media Type: Generic
Protocol: USB
SMART Status: Not Supported
Volume UUID: FF6E9075-BB92-4631-81DA-3481B13C2470
Disk / Partition UUID: FF6E9075-BB92-4631-81DA-3481B13C2470
Disk Size: 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
Device Block Size: 4096 Bytes
Container Total Space: 499.9 GB (499898105856 Bytes) (exactly 976363488 512-Byte-Units)
Container Free Space: 104.8 GB (104797212672 Bytes) (exactly 204682056 512-Byte-Units)
Allocation Block Size: 4096 Bytes
Media OS Use Only: No
Media Read-Only: No
Volume Read-Only: No
Device Location: External
Removable Media: Fixed
Solid State: Yes
This disk is an APFS Volume. APFS Information:
APFS Container: disk3
APFS Physical Store: disk2s2
Fusion Drive: No
FileVault: Yes
Sealed: No
Locked: No
I'm running into the same/similar issue.
Been through uninstall procedure (https://github.com/NixOS/nix/pull/6144) about 5 times.
Booting from external SSD, but not using FileVault. macOS Catalina.
Referenced from: /nix/var/nix/profiles/default/bin/nix-daemon
Reason: no suitable image found. Did find:
file system sandbox blocked open() of '/nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib'
/nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib: stat() failed with errno=1
file system sandbox blocked open() of '/nix/store/jd1l64vr0b5y1qvdsrminn53gkvamm32-editline-1.17.1/lib/libeditline.1.dylib'```
Still just kinda guessing/collecting.
I don't recall any ~sandbox reports like this before, but the external disk bit does remind me a little of some issues related to Nix installs on VMs and EC2. We don't fully understand it, but macOS does treat internal and external drives a little differently, so I guess that's the thread to pull on for now.
diskutil info -plist /nixsudo vsdbutil -c /sysadminctl -secureTokenStatus $USER
It might also be worth opening Console.app to see if there are any obviously-correlated messages that give a little more insight.
The darwin-store launch daemon seems to fail each boot. The following is after I run it manually with sudo launchctl kickstart -k system/org.nixos.darwin-store.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>APFSContainerFree</key>
<integer>134000287744</integer>
<key>APFSContainerReference</key>
<string>disk3</string>
<key>APFSContainerSize</key>
<integer>499898105856</integer>
<key>APFSPhysicalStores</key>
<array>
<dict>
<key>APFSPhysicalStore</key>
<string>disk2s2</string>
</dict>
</array>
<key>APFSSnapshot</key>
<false/>
<key>Bootable</key>
<true/>
<key>BooterDeviceIdentifier</key>
<string>disk3s2</string>
<key>BusProtocol</key>
<string>USB</string>
<key>CanBeMadeBootable</key>
<false/>
<key>CanBeMadeBootableRequiresDestroy</key>
<false/>
<key>Content</key>
<string>41504653-0000-11AA-AA11-00306543ECAC</string>
<key>DeviceBlockSize</key>
<integer>4096</integer>
<key>DeviceIdentifier</key>
<string>disk3s7</string>
<key>DeviceNode</key>
<string>/dev/disk3s7</string>
<key>DeviceTreePath</key>
<string>IODeviceTree:/PCI0@0/PEG2@1,2/UPSB@0/DSB2@2/XHC3@0</string>
<key>DiskUUID</key>
<string>FF6E9075-BB92-4631-81DA-3481B13C2470</string>
<key>Ejectable</key>
<true/>
<key>EjectableMediaAutomaticUnderSoftwareControl</key>
<false/>
<key>EjectableOnly</key>
<false/>
<key>Encryption</key>
<true/>
<key>EncryptionThisVolumeProper</key>
<true/>
<key>FileVault</key>
<true/>
<key>FilesystemName</key>
<string>APFS</string>
<key>FilesystemType</key>
<string>apfs</string>
<key>FilesystemUserVisibleName</key>
<string>APFS</string>
<key>FreeSpace</key>
<integer>0</integer>
<key>Fusion</key>
<false/>
<key>GlobalPermissionsEnabled</key>
<true/>
<key>IOKitSize</key>
<integer>499898105856</integer>
<key>IORegistryEntryName</key>
<string>Nix Store</string>
<key>Internal</key>
<false/>
<key>Locked</key>
<false/>
<key>MediaName</key>
<string></string>
<key>MediaType</key>
<string>Generic</string>
<key>MountPoint</key>
<string>/nix</string>
<key>OSInternalMedia</key>
<false/>
<key>ParentWholeDisk</key>
<string>disk3</string>
<key>PartitionMapPartition</key>
<false/>
<key>RAIDMaster</key>
<false/>
<key>RAIDSlice</key>
<false/>
<key>RecoveryDeviceIdentifier</key>
<string>disk3s3</string>
<key>Removable</key>
<false/>
<key>RemovableMedia</key>
<false/>
<key>RemovableMediaOrExternalDevice</key>
<true/>
<key>SMARTDeviceSpecificKeysMayVaryNotGuaranteed</key>
<dict/>
<key>SMARTStatus</key>
<string>Not Supported</string>
<key>Sealed</key>
<string>No</string>
<key>Size</key>
<integer>499898105856</integer>
<key>SolidState</key>
<true/>
<key>SupportsGlobalPermissionsDisable</key>
<true/>
<key>SystemImage</key>
<false/>
<key>TotalSize</key>
<integer>499898105856</integer>
<key>VolumeAllocationBlockSize</key>
<integer>4096</integer>
<key>VolumeName</key>
<string>Nix Store</string>
<key>VolumeSize</key>
<integer>0</integer>
<key>VolumeUUID</key>
<string>FF6E9075-BB92-4631-81DA-3481B13C2470</string>
<key>WholeDisk</key>
<false/>
<key>Writable</key>
<true/>
<key>WritableMedia</key>
<true/>
<key>WritableVolume</key>
<true/>
</dict>
</plist>
MacBook-Pro% sudo vsdbutil -c /
No entry found for '/'.
MacBook-Pro% sysadminctl -secureTokenStatus $USER
2022-03-27 08:53:48.050 sysadminctl[5332:334253] Secure token is ENABLED for user NN
I have no experience at all with sandboxing on MacOS, except as an end user. However, something that showed up when I searched was that there seems to be two different privileges, SystemPolicyAllFiles and SystemPolicyRemovableVolumes, and that despite the naming (and description) the first one does not overlap with the second, e.g. both are needed. Is nix-daemon asking for both?
In my System preferences, under Security and Integrity -> Integrity -> Files and Folders, Terminal is listed as having "Removable Volumes", while nix-daemon is listed with "Full Disk Access". (I'm using Swedish so the terminology might be a bit different.) Maybe nix-daemon inherits the Terminal-privileges when I execute it manually?
Since we're exploring I'll try to think out loud for posterity (don't worry too much if something I say doesn't make sense; feel free to ask questions). I'll start with the command outputs (leaving your followup comment on the privileges aside for now):
-
I haven't been through this in detail yet, but the one item I was looking for here looks OK:
<key>GlobalPermissionsEnabled</key> <true/> -
This one might be pointing to the key difference. My device (on the internal SSD) yields
Permissions on '/' are enabled. -
This looks OK as well.
The difference I note in 2 is out in that territory where we don't exactly understand what the system is doing well enough to be sure if it's a problem or not.
- Our working thesis has been that we just need
GlobalPermissionsEnabledto be true for the Nix volume, but perhaps the root volume also has to have the same. - We might be able to use
vsdbutilto change this and see if it fixes the issue, but we're leaving that as a last resort because the manpage also indicates thatvsdbutilis deprecated and tells us to use fstab instead. - instead we:
So my thought for the moment is:
-
Before we try anything, confirm that the value is
falsefor theGlobalPermissionsEnabledkey in the following command:diskutil info -plist / -
If the value is indeed
false, try running/usr/sbin/diskutil enableOwnership /and then see if the Nix volume mounts on reboot. (If this messes anything else up, you should be able to use/usr/sbin/diskutil disableOwnership /to reverse it.)
MacBook-Pro% diskutil info -plist / | grep -A1 GlobalPermissions
<key>GlobalPermissionsEnabled</key>
<true/>
--
<key>SupportsGlobalPermissionsDisable</key>
<true/>
So no use running enableOwnership, then?
Edit: Posted the wrong command execution. Output still the same.
Hmm. I was thinking not, but I just double-checked the manpage for diskutil, and it says enableOwnership will:
Enable ownership of a volume. The on-root-disk Volume Database at /var/db/volinfo.database is manipulated such that the User and Group ID settings of files, directories, and links (file system objects, or "FSOs") on the target volume are taken into account.
That's the same database vsdbutil would modify. We thought there was a good correlation between GlobalPermissionsEnabled == false and the need to run it, but that may be wrong.
So--yeah, go ahead and try sudo /usr/sbin/diskutil enableOwnership /
No difference after sudo /usr/sbin/diskutil enableOwnership / and reboot. The darwin-store service failed to mount the nix-volume until I launched it manually. And after that the same error in nix-daemon.log.
Since I had GlobalPermissionsEnabled before there should be no obvious need to reverse anything from the enableOwnership command?
Drat. Does sudo vsdbutil -c / now report Permissions on '/' are enabled.? And What does sudo vsdbutil -c /nix report?
MacBook-Pro% sudo vsdbutil -c /
Permissions on '/' are enabled.
MacBook-Pro% sudo vsdbutil -c /nix
No entry found for '/nix'.
Hmm. Can you try the enablePermissions command for /nix as well (and reboot)? I assumed the installer would have done that, but perhaps not.
I just did (including reboot), but still no difference, except for the output of sudo vsdbutil -c /nix which now reports Permissions on '/nix' are enabled.
I have no idea if it could have any impact, but I failed to mention when opening the issue that I had nix installed prior to upgrading to Monterey. So the re-installation failed asking me to remove nix entries in bashrc etc. I did all these things (repeatedly) until the installation went through without errors. While I removed the things I was asked to, I guess that e.g. build users created by the earlier installation might have been reused. (I did remove the new nix volume between each retry.)
Ah. Perhaps. A complete uninstall is tricky and might be confounding. Were you on Catalina? It was working with the same general setup?
It might be worth following the not-quite-released uninstall directions from the unstable manual https://nixos.org/manual/nix/unstable/installation/installing-binary.html#macos and giving a reinstall a try.
I was not on Catalina but went straight from Mojave to Monterey.
I now performed all the steps from the linked uninstall directions, followed by a reinstall of nix. The issue persists. The volume was mounted after installation but nix-daemon fails to access libsodium during so is not running. Will reboot now.
No luck. No mounted volume, and after manually triggering darwin-store, nix-daemon still fails.
Ok. Makes sense that you leapfrogged Catalina.
Back to your permission note earlier, nix doesn't currently use any of the ~Security permissions (and I don't think it can, automatically--IIUC the best anyone can do is trigger an action that would cause their app to show up in the list and then open the panel for the user to toggle it).
I would guess that you have added both of these permissions over the years while troubleshooting nix or other software, and the uninstall won't flush them. I think I recall setting full disk for the daemon being a workaround that the same set of EC2/VM users were using to get nix working before the more recent enableOwnership change (which at least appears to have caused the trickle of those reports to dry up).
Given that history, it does make sense to me that adding some may be forcing the issue. If you don't already have it installed with the same permission, it might be interesting to install iterm2 and see if manually invoking the daemon from there triggers the sandbox error, or if it just works?
Fair warning that I am otherwise running low on actionable hunches. I'll try to do a little more reading myself, but my responses may slow down a little if we hit another dead-end or two.
Thank you so much for all your effort so far! I wouldn't have expected anywhere near this engagement for an edge case like mine, but your response has been better than any paid customer service I have ever used ...
I would also like to thank you and your fellow committers for your work on the installer. It has become really great, with clear information on what's going on and helpful instructions when it detects something's off. It so good I almost don't mind having to reinstall nix for the fiftyeleventh time ...
I will test with iTerm2, but probably not before the weekend.
Hitting the same issue. Manually running sudo -i /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon does seem to fix it. Anyone know the root cause here?
I'm seeing this on osx 12.4, running on an amazon mac2.metal instance (perhaps that's a way that people can repro this).
Hitting the same issue. Manually running
sudo -i /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemondoes seem to fix it.
As in, running it once permanently fixes it? Or just that it works that time?
It just works that time. So I leave that terminal window open to run the daemon, then in other terminal windows my nix-shell commands will work (I can see the daemon terminal outputting 'accepted connection').
What do stat /nix and stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon say?
$ stat /nix
16777241 2 drwxr-xr-x 5 root nixbld 0 160 "Jul 26 23:32:17 2022" "Jul 26 23:15:51 2022" "Jul 26 23:15:51 2022" "Jul 26 23:15:33 2022" 4096 0 0 /nix
$ stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon
16777241 616 lr-xr-xr-x 1 ec2-user staff 0 3 "Jul 26 23:15:28 2022" "Jan 1 00:00:01 1970" "Jul 26 23:15:52 2022" "Jan 1 00:00:01 1970" 4096 0 0 /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix-daemon
$ stat /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix
16777241 615 -r-xr-xr-x 1 ec2-user staff 0 3061456 "Jul 27 05:33:43 2022" "Jan 1 00:00:01 1970" "Jul 26 23:15:52 2022" "Jan 1 00:00:01 1970" 4096 5984 0 /nix/store/gp7k17iy1n7hgf97qwnxw28c6v9nhb1i-nix-2.10.3/bin/nix
Two thoughts:
- starting to wonder if this is a different (or maybe more modern?) manifestation of the problem reposted in #4640.
- I doubt
nixbldshould be the group for/nix. I think mayberootis also a sign of trouble, per #4640, though I've overwithdrawn my brain bank for the day. I have an older install that I can't readily use as a reference for what yours should look like, but I'll ask on Matrix and see if someone with a working modern install can chime in...
I have a physical mac where nix works fine (the above output is from a mac VM on ec2). Stat'ing the same files, I see on my working physical mac that root.nixbld owns /nix and everything in it.
So I think the problem on the mac VM is that the logged-in user is somehow gaining ownership of some files in the /nix/store (or perhaps the inverse like you said: the logged-in user should own everything, not root). Although it's unclear how this relates to the root cause problem of the fs sandbox blocking open() in /nix. Tomorrow, I'll try to manually chown /nix and see if it fixes the issue.
Someone on matrix chimed in and said they do have root/nixbld for both /nix and a copy of nix-daemon.
The source indicates that root/nixbld are correct.
(Someone on an older macos said they have root/wheel. Not sure what the cause of that one is...)
It's possible the installer is holding something wrong. I know it uses sudo a lot and I gather macos sudo is a little weird.
If basically everything else in your store is owned by your user (and the same isn't true on your physical Mac), I'd wonder first if aws is using a modified sudo that breaks our assumptions?
I've ran into the same issue on macOS Monterey 12.5: I moved /nix to external disk and re-juggled mount points, so that external disk was mounted as /nix. After that nix-daemon refused to start with file system sandbox blocked open() for libsodium:
% sudo launchctl debug system/org.nixos.nix-daemon --stderr
Service configured for next launch.
dyld[59104]: Library not loaded: '/nix/store/3g5jq3wxcna446klkmrvvkbjzcy6ppcx-libsodium-1.0.18/lib/libsodium.23.dylib'
Referenced from: '/nix/store/qzran14w2x48nk5dp0pji5rw0xj94r7s-nix-2.8.1/bin/nix'
Reason: tried: '/nix/store/3g5jq3wxcna446klkmrvvkbjzcy6ppcx-libsodium-1.0.18/lib/libsodium.23.dylib' (file system sandbox blocked open()), '/usr/local/lib/libsodium.23.dylib' (no such file), '/usr/lib/libsodium.23.dylib' (no such file)
When I granted "Full Disk Access" to sh, this error disappeared and the daemon started just fine. Doing the same for nix binary didn't yield any changes, so I assume that the sandbox is built based on the binary that starts the daemon, not on the one that it exec()'ed into.
I wasn't happy with giving any user access to an executable with full disk access, so I:
- made a copy of
/bin/sh:sudo mkdir -p /usr/local/bin && sudo cp /bin/sh /usr/local/bin/shWithFullDiskAccess - allowed only root to run it:
sudo chmod 400 /usr/local/bin/shWithFullDiskAccess - added it to "Full Disk Access" in "Privacy & Policy" settings tab
- replaced
/bin/shwith/usr/local/bin/shWithFullDiskAccessin/Library/LaunchDaemons/org.nixos.nix-daemon.plist
After all this, nix-daemon is running fine. It would be nice to have a small "starter" binary for nix-daemon that we could place on root partition instead though.
@YorikSar Am I understanding right that this is on a local system that's easy to poke at? (I don't have fresh ideas atm; just asking for the record since people able/willing to poke at this kind of issue on-demand have been scarce.)