nix-installer icon indicating copy to clipboard operation
nix-installer copied to clipboard

"launchctl" "bootout" "system" "/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist" Boot-out failed: 5: Input/output error

Open szicari-streambit opened this issue 1 year ago • 13 comments

Error

Error: 
   0: Install failure
   1: Error executing action
   2: Action `create_nix_hook_service` errored
   3: Failed to execute command with status 5 `"launchctl" "bootout" "system" "/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist"`, stdout: 
      stderr: Boot-out failed: 5: Input/output error


Metadata

key value
version 0.26.1
os macos
arch aarch64

szicari-streambit avatar Sep 20 '24 16:09 szicari-streambit

I'd like to note that I can get the installation to succeed if I allow the script to undo what it tried to do, restart my Mac, and then run the script as one of the first things I do after logging in.

szicari-streambit avatar Sep 20 '24 16:09 szicari-streambit

That's interesting... If you see this again, I'd be interested in the output of launchctl print system/systems.determinate.nix-installer.nix-hook.

(I wish launchctl was more verbose with what actually went wrong)

cole-h avatar Sep 20 '24 16:09 cole-h

Ok. I will post the result here if I have to reinstall again.

szicari-streambit avatar Sep 20 '24 19:09 szicari-streambit

@cole-h this is how it looks for me (with same error happening, albeit still on Sonoma):

launchctl print system/systems.determinate.nix-installer.nix-hook
system/systems.determinate.nix-installer.nix-hook = {
        active count = 0
        path = /Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist
        type = LaunchDaemon
        state = not running

        program = /bin/sh
        arguments = {
                /bin/sh
                -c
                /bin/wait4path /nix/nix-installer && /nix/nix-installer repair
        }

        stdout path = /nix/.nix-installer-hook.out.log
        stderr path = /nix/.nix-installer-hook.err.log
        default environment = {
                PATH => /usr/bin:/bin:/usr/sbin:/sbin
        }

        environment = {
                XPC_SERVICE_NAME => systems.determinate.nix-installer.nix-hook
        }

        domain = system
        minimum runtime = 10
        exit timeout = 5
        runs = 2
        last exit code = 0

        semaphores = {
                successful exit => 0
        }

        spawn type = daemon (3)
        jetsam priority = 40
        jetsam memory limit (active) = (unlimited)
        jetsam memory limit (inactive) = (unlimited)
        jetsamproperties category = daemon
        jetsam thread limit = 32
        cpumon = default
        probabilistic guard malloc policy = {
                activation rate = 1/1000
                sample rate = 1/0
        }

        properties = inferred program | managed LWCR | has LWCR
}

kblcuk avatar Sep 21 '24 08:09 kblcuk

Thanks! Does the plist path in that output exist as a file...?

cole-h avatar Sep 21 '24 18:09 cole-h

I actually got it fixed by reboot 😅 And now that I tried uninstalling / reinstalling nix a couple of times, I don't seem to get that problem anymore.

kblcuk avatar Sep 22 '24 11:09 kblcuk

That's interesting... If you see this again, I'd be interested in the output of launchctl print system/systems.determinate.nix-installer.nix-hook.

(I wish launchctl was more verbose with what actually went wrong)

Command and output:

$ launchctl print system/systems.determinate.nix-installer.nix-hook
system/systems.determinate.nix-installer.nix-hook = {
	active count = 1
	path = /Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist
	type = LaunchDaemon
	state = running

	program = /bin/sh
	arguments = {
		/bin/sh
		-c
		/bin/wait4path /nix/nix-installer && /nix/nix-installer repair
	}

	stdout path = /nix/.nix-installer-hook.out.log
	stderr path = /nix/.nix-installer-hook.err.log
	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		XPC_SERVICE_NAME => systems.determinate.nix-installer.nix-hook
	}

	domain = system
	minimum runtime = 10
	exit timeout = 5
	runs = 124
	pid = 6074
	immediate reason = semaphore
	forks = 1
	execs = 2
	initialized = 1
	trampolined = 1
	started suspended = 0
	proxy started suspended = 0
	last exit code = 78: EX_CONFIG

	semaphores = {
		successful exit => 0
	}

	spawn type = daemon (3)
	jetsam priority = 40
	jetsam memory limit (active) = (unlimited)
	jetsam memory limit (inactive) = (unlimited)
	jetsamproperties category = daemon
	jetsam thread limit = 32
	cpumon = default
	probabilistic guard malloc policy = {
		activation rate = 1/1000
		sample rate = 1/0
	}

	properties = inferred program | managed LWCR | has LWCR
}

Also, not sure how relevant, but I've had to install / uninstall / reboot / install Nix on every reboot so far. It hasn't "stayed" installed for whatever reason. The error that leads to the above reinstall cycle is because the Nix suite of commands aren't found and then the install fails because /nix is readonly. It's a vicious cycle. 😊

szicari-streambit avatar Sep 23 '24 14:09 szicari-streambit

@szicari-streambit Does /Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist exist when you run into this failure? Is there anything (interesting) in /nix/.nix-installer-hook.out.log or /nix/.nix-installer-hook.err.log? What does ps aux | grep nix-installer show, if anything?

Also, not sure how relevant, but I've had to install / uninstall / reboot / install Nix on every reboot so far. It hasn't "stayed" installed for whatever reason. The error that leads to the above reinstall cycle is because the Nix suite of commands aren't found and then the install fails because /nix is readonly.

That's certainly strange, but probably bears investigation in another ticket. (A few things to try before that: does starting up a new terminal window make the nix commands show up? What if you ssh into the machine, if you're able to set that up? And finally: what if you run /nix/nix-installer repair hooks and try my previous 2 suggestions again?)

cole-h avatar Sep 23 '24 14:09 cole-h

Thanks for all this. I have my task list for the next time this occurs.

szicari-streambit avatar Sep 23 '24 14:09 szicari-streambit

I had the same issue on macOS 15.0 (24A335). Installing immediately after a reboot fixed the issue for me as well.

jhult avatar Sep 23 '24 18:09 jhult

Good morning, @cole-h. After shutting down my computer, I have again found Nix unavailable. The result of your steps above:

After the first failed install, neither of the log files were present:

Screenshot 2024-09-25 at 08 48 13

The plist file was also not present.

Screenshot 2024-09-25 at 08 49 40

Here is the full output from the failure:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
info: downloading installer https://install.determinate.systems/nix/tag/v0.26.2/nix-installer-aarch64-darwin
 INFO nix-installer v0.26.2
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
 INFO nix-installer v0.26.2
Nix install plan (v0.26.2)
Planner: macos (with default settings)

Planned actions:
* Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
* Extract the bundled Nix (originally from /nix/store/6kl4439887d6ggjgrc41h3jqrcvbwlki-nix-binary-tarball-2.24.7/nix-2.24.7-aarch64-darwin.tar.xz)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Create build users (UID 351-382) and group (GID 350)
* Configure Time Machine exclusions
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Configuring zsh to support using Nix in non-interactive shells
* Unload, then recreate a `launchctl` plist to put Nix into your PATH
* Configure upstream Nix daemon service
* Remove directory `/nix/temp-install-dir`


Proceed? ([Y]es/[n]o/[e]xplain):
 INFO Step: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
 INFO Step: Provision Nix
 INFO Step: Create build users (UID 351-382) and group (GID 350)
 INFO Step: Configure Time Machine exclusions
 INFO Step: Configure Nix
 INFO Step: Configuring zsh to support using Nix in non-interactive shells
 INFO Step: Unload, then recreate a `launchctl` plist to put Nix into your PATH
ERROR
   0: Install failure
   1: Error executing action
   2: Action `create_nix_hook_service` errored
   3: Failed to execute command with status 5 `"launchctl" "bootout" "system" "/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist"`, stdout:
      stderr: Boot-out failed: 5: Input/output error

   3:

Location:
   src/cli/subcommand/install.rs:253

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Install+failure%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60create_nix_hook_service%60+errored%0A+++3%3A+Failed+to+execute+command+with+status+5+%60%22launchctl%22+%22bootout%22+%22system%22+%22%2FLibrary%2FLaunchDaemons%2Fsystems.determinate.nix-installer.nix-hook.plist%22%60%2C+stdout%3A+%0A++++++stderr%3A+Boot-out+failed%3A+5%3A+Input%2Foutput+error%0A%0A%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.26.2%7C%0A%7C**os**%7Cmacos%7C%0A%7C**arch**%7Caarch64%7C%0A
Installation failure, offering to revert...
Nix uninstall plan (v0.26.2)

Planner: macos (with default settings)

Planned actions:
* Delete file `/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist`
* Remove the Nix configuration from zsh's non-login shells
* Unconfigure the shell profiles
* Remove the Nix configuration in `/etc/nix/nix.conf`
* Unset the default Nix profile
* Remove time machine exclusions
* Remove Nix users and group
* Remove the directory tree in `/nix`
* Remove the APFS volume `Nix Store` on `disk3`


Proceed? ([Y]es/[n]o/[e]xplain):
 INFO Revert: Remove directory `/nix/temp-install-dir`
 INFO Revert: Configure upstream Nix daemon service
 INFO Revert: Unload, then recreate a `launchctl` plist to put Nix into your PATH
 INFO Revert: Configuring zsh to support using Nix in non-interactive shells
 INFO Revert: Configure Nix
 INFO Revert: Configure Time Machine exclusions
 INFO Revert: Create build users (UID 351-382) and group (GID 350)
 INFO Revert: Provision Nix
 INFO Revert: Create an APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
Error:
   0: Multiple errors

Location:
   src/cli/subcommand/install.rs:285

Error:
   0: Action `create_nix_hook_service` errored
   1: Remove file `/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist`
   2: No such file or directory (os error 2)

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Multiple+errors%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.26.2%7C%0A%7C**os**%7Cmacos%7C%0A%7C**arch**%7Caarch64%7C%0A

A new terminal did not cause the nix commands to be present.

I'm not sure if this is relevant, but I see the Nix volume was attached as expected when the system booted up.

Screenshot 2024-09-25 at 08 37 11

szicari-streambit avatar Sep 25 '24 13:09 szicari-streambit

@szicari-streambit That is intriguing... Thank for the info! I think we may just need to check if that file exists before trying to do any launchctl funsies on it. Is it possible that trying to install again works for you in the interim? If not, I'll write a patch for you to try ASAP.

Additionally, is this an MDM-managed device? I'm having a hard time figuring out why Nix would just disappear on reboot, since we have that hook (when it successfully installs....). Next time that happens, I'd be interested in seeing your /etc/zshrc and /etc/bashrc, if you wouldn't mind uploading them to a gist or something. That's where we should be sourcing the script that tells the shells "here's where to look for Nix binaries and other stuff!"...

cole-h avatar Sep 26 '24 14:09 cole-h

@cole-h I just booted up without Nix once again. I checked and found nothing where /nix should be.

ls -lah /nix
total 0
drwxr-xr-x   2 root  wheel    64B Sep 30 09:58 .
drwxr-xr-x  20 root  wheel   640B Sep  5 15:54 ..

After a bit of digging, I found Nix was mounted at /Volumes/Nix Store:

ls -lah /Volumes/Nix\ Store/
total 121968
drwxr-xr-x@    9 root        wheel    288B Sep 25 09:02 .
drwxr-xr-x     7 root        wheel    224B Sep 30 10:03 ..
drwx------     4 root        wheel    128B Sep 25 09:02 .Spotlight-V100
d-wx--x--t     3 root        wheel     96B Sep 25 09:02 .Trashes
drwx------   429 root        wheel     13K Sep 30 10:03 .fseventsd
-rwxr-xr-x     1 seanzicari  staff     60M Sep 25 09:02 nix-installer
-rw-r--r--     1 root        wheel     38K Sep 25 09:02 receipt.json
drwxrwxr-t@ 6346 root        nixbld   198K Sep 26 10:34 store
drwxr-xr-x@    4 root        wheel    128B Sep 25 09:02 var

I remounted the Nix volume in the right place:

# Unmount command not shown but was executed here

diskutil mount -mountPoint /nix disk3s7
Volume Nix Store on disk3s7 mounted

Upon opening a new shell, I found all my Nix and Nix-installed tooling was restored! So it seems my issue is the Nix Store is mounted in the wrong location on startup.

I have very little familiarity with how macOS allows configuration of mounted volumes on bootup, but I saw the following in /etc/fstab:

#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#








# nix-installer created volume labelled `Nix Store`

So maybe something in the automount configuration wasn't completed properly during install?

szicari-streambit avatar Sep 30 '24 15:09 szicari-streambit

Also hitting this. Originally manifested as home-manager not being able to connect to the nix socket (possibly after a background system update, unsure).

  • Rebooted
  • uninstalled nix with /nix/nix-installer uninstall
  • Installed nix with curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --determinate
ERROR
   0: Install failure
   1: Error executing action
   2: Action `create_nix_hook_service` errored
   3: Failed to execute command with status 5 `"launchctl" "bootout" "system" "/Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist"`, stdout:
      stderr: Boot-out failed: 5: Input/output error

Inspecting launchctl launchctl print system/systems.determinate.nix-installer.nix-hook

system/systems.determinate.nix-installer.nix-hook = {
	active count = 0
	path = /Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist
	type = LaunchDaemon
	state = not running

	program = /bin/sh
	arguments = {
		/bin/sh
		-c
		/bin/wait4path /nix/nix-installer && /nix/nix-installer repair
	}

	stdout path = /nix/.nix-installer-hook.out.log
	stderr path = /nix/.nix-installer-hook.err.log
	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		XPC_SERVICE_NAME => systems.determinate.nix-installer.nix-hook
	}

	domain = system
	minimum runtime = 10
	exit timeout = 5
	runs = 2
	last exit code = 0

	semaphores = {
		successful exit => 0
	}

	spawn type = daemon (3)
	jetsam priority = 40
	jetsam memory limit (active) = (unlimited)
	jetsam memory limit (inactive) = (unlimited)
	jetsamproperties category = daemon
	jetsam thread limit = 32
	cpumon = default
	probabilistic guard malloc policy = {
		activation rate = 1/1000
		sample rate = 1/0
	}

	properties = inferred program | managed LWCR | has LWCR
}


- does nix-hook.plist exist? No

stat: /Library/LaunchDaemons/systems.determinate.nix-installer.nix-hook.plist: stat: No such file or directory


strangemonad avatar Oct 30 '24 21:10 strangemonad

I was actually able to reproduce the bootout issue the other day (as noted in my comment on another issue: https://github.com/DeterminateSystems/nix-installer/issues/1230#issuecomment-2442591279). I have a PR up at https://github.com/DeterminateSystems/nix-installer/pull/1257 that I think fixes this. In the meantime, it also has a suggestion to run if you run into this issue on install (reposting here):

sudo launchctl bootout system/systems.determinate.nix-installer.nix-hook

cole-h avatar Oct 30 '24 21:10 cole-h