Fleet icon doesn't appear in menu bar on macOS 14.7

[noahtalerman]

UPDATE: Re-opening bug as we have a report of this on 14.7.

UPDATE: Closed because we couldn't reproduce. More details here (noahtalerman 2024-06-05)

UPDATE: After restarting my computer, the Fleet icon appeared and I could access Fleet Desktop. I think it's worth time-boxing this one to see if there's anything reproducible in my logs (noahtalerman 2024-05-24)

Fleet version: Observed in Fleet's dogfood environment (4.49.4). fleetd 1.24 (osquery: 5.12.2, Orbit: 1.24.0, Fleet Desktop: 1.24.0)

Web browser and operating system: macOS 14.5

---

💥  Actual behavior

Summary from debug session with prospect on macOS 14.7 running on x86:

orbit errors seen:

2024-10-11T10:20:51-04:00 INF opening path="/opt/orbit/bin/desktop/macos/stable/Fleet Desktop.app"
The application cannot be opened for an unexpected reason, error=Error Domain=NSOSStatusErrorDomain Code=-10822 "kLSServerCommunicationErr: The server process (registration and recent items) is not available" UserInfo={_LSLine=517, _LSFunction=-[_LSRemoteOpenCall invokeWithError:]}

system log errors seen:

error   15:04:44.688247-0400    open    Process unable to access CoreServicesUIAgent via kLSCoreServicesUIAgentXPCName name.
error   15:04:44.690541-0400    open    -[_LSRemoteOpenCall invokeWithError:]: XPC error encountered talking to CSUIA: <dictionary: 0x7ff8427b7cc0> { count = 1, transaction: 0, voucher = 0x0, contents =
    "XPCErrorDescription" => <string: 0x7ff8427b7e58> { length = 18, contents = "Connection invalid" }
}

The following actions did not work:

• sudo launchctl kickstart -k system/com.fleetdm.orbit
• uninstalling and reinstalling
• installing a package built on a different machine
• installing a package built with --disable-keystore flag, to rule out keychain

The following did work:

• Manually launching fleet desktop with proper env vars: open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app
• Restarting the machine

Potential next steps:

• Provide an easy way for user/admin to manually launch Fleet Desktop if it does not come up
• Explore other ways we can launch Fleet Desktop, maybe like:

launchctl asuser $uid sudo -u "$currentUser" /usr/bin/open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app

• Reference: https://scriptingosx.com/2020/08/running-a-command-as-another-user/
• Example: https://github.com/bvanpeski/unActivationLock/pull/12/files#diff-04ce03e58be7bf3ed9bdaeb13cdce7c785148cb5142dc58811104563b0e97767

[sharon-fdm]

Need to root cause. Estimating at 3 but could be more depending on the bug.

[lucasmrod]

@noahtalerman

I can see the following error in the attached logs. All the Fleet Desktop errors happening on May 21st (day of the upgrade):

11983040 The application cannot be opened for an unexpected reason, error=Error Domain=NSOSStatusErrorDomain Code=-10822 "kLSServerCommunicationErr: The server process (registration and recent items) is not available" UserInfo={_LSLine=517, _LSFunction=-[_LSRemoteOpenCall invokeWithError:]}
11983041 2024-05-21T10:12:41-04:00 DBG execuser.Run error="open path \"/opt/orbit/bin/desktop/macos/edge/Fleet Desktop.app\": exit status 1"

The above error log happened 228 times (because orbit tries to start Fleet Desktop every 30s if it's not running).

I was not able to reproduce on a macOS Parallels VM upgrading from 13.X to 14.5. The Fleet Desktop icon showed up after the upgrade.

Maybe we can close as "can't repro" for now and re-open if it happens to someone else? (after posting the error log in this comment it should show up on Github search). (I'm also adding the :reproduce label too.)

Let me know your thoughts.

[noahtalerman]

Maybe we can close as "can't repro" for now and re-open if it happens to someone else?

Sounds good to me. Thanks Lucas!

[fleet-release]

In the cloud city, Fleet icon finds its own way, After system plays.

Upgrade brings challenge, Yet, a reboot gives new life, Access now displayed.

In this dance of tech, Nature's cycle is mirrored, Rebirth after rest.

[roperzh]

This is happening to me after fleetd updated to 1.26.0, happy to provide any details

 $ /opt/orbit/bin/orbit/orbit --version
orbit 1.26.0

$ /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app/Contents/MacOS/fleet-desktop --version
fleet-desktop 1.26.0

I see this in the logs:

The application cannot be opened for an unexpected reason, error=Error Domain=NSOSStatusErrorDomain Code=-10822 "kLSServerCommunicationErr: The server process (registration and recent items) is not available" UserInfo={_LSLine=517, _LSFunction=-[_LSRemoteOpenCall invokeWithError:]}

It was pointed by Jazhiel that we saw this exact same error before in https://github.com/fleetdm/fleet/pull/16090, but I find it weird that I'm logged in and it's still happening, which hints that something else is going on.

[lucasmrod]

Thanks Roberto! As promised I'll re-open.

Here's @gillespi314's comment from the Slack thread:

I think it happens when there is a TUF update received while the user is not logged in

Worth testing that scenario.

[gillespi314]

Potentially related: https://github.com/fleetdm/fleet/issues/19499

[lucasmrod]

@noahtalerman Was your device enrolled to Fleet MDM manually or automatic?

Am still trying to reproduce. Current theory is that it happens only on M1. Roberto, you and (possibly) Kathy hit this bug (and the three of you are using M1, whereas I've been trying to reproduce on Intel)

[noahtalerman]

Hey @lucasmrod! I turned on MDM manually. My workstation isn't in ABM.

[lucasmrod]

We still need to reproduce reliably...

Adding a few more details of the folks that reproduced the issue:

• Roberto: M1, DEP enrolled device. Restarted fleetd without success, only a restart of the OS fixed the issue.
• Noah: M1, manual enroll, only a restart fixed the issue.

[zayhanlon]

@sharon-fdm this is being reported now by a community user on macos version 14.4.1 as well, not just 14.5

Here is some new information: Happened with first time fleet enrolls

• Device running 14.4.1, required the user to have to restart their machine 3 times before Fleet Desktop launched in the menu bar.
• Device running 14.5, required the user a restart to get Fleet Desktop to show up in the menu bar.

For a new hire, we had to tailor the instruction, after they run the fleet installer, we ask them to restart their machines because Fleet will 100% not launch. But then some users after doing the restart will still be unable to launch Fleet Deskop

[lucasmrod]

https://github.com/fleetdm/fleet/issues/19499#issuecomment-2273231223

[sharon-fdm]

https://fleetdm.slack.com/archives/C01EZVBHFHU/p1722972217951119

[getvictor]

Does restarting orbit also work (instead of restarting OS): sudo launchctl kickstart -k system/com.fleetdm.orbit?

[roperzh]

Victor: I didn't try with launchctl but I did try sudo killall orbit, which kicked a restart and it didn't fix it.

[getvictor]

If restarting doesn't work, it suggests the issue may be outside the app. Maybe with launch services or with user permissions.

Have we seen this on 14.6 as well?

[getvictor]

Closing this. Please reopen if this is seen with 14.6 or later.

[fleet-release]

Restart solves the glitch, Fleet icon now takes its niche, Smooth as river's stitch.

[lukeheath]

Re-opening bug as we have a report of this on 14.7.

[getvictor]

Summary from debug session with prospect on macOS 14.7 running on x86:

orbit errors seen:

2024-10-11T10:20:51-04:00 INF opening path="/opt/orbit/bin/desktop/macos/stable/Fleet Desktop.app"
The application cannot be opened for an unexpected reason, error=Error Domain=NSOSStatusErrorDomain Code=-10822 "kLSServerCommunicationErr: The server process (registration and recent items) is not available" UserInfo={_LSLine=517, _LSFunction=-[_LSRemoteOpenCall invokeWithError:]}

system log errors seen:

error   15:04:44.688247-0400    open    Process unable to access CoreServicesUIAgent via kLSCoreServicesUIAgentXPCName name.
error   15:04:44.690541-0400    open    -[_LSRemoteOpenCall invokeWithError:]: XPC error encountered talking to CSUIA: <dictionary: 0x7ff8427b7cc0> { count = 1, transaction: 0, voucher = 0x0, contents =
    "XPCErrorDescription" => <string: 0x7ff8427b7e58> { length = 18, contents = "Connection invalid" }
}

The following actions did not work:

• sudo launchctl kickstart -k system/com.fleetdm.orbit
• uninstalling and reinstalling
• installing a package built on a different machine
• installing a package built with --disable-keystore flag, to rule out keychain

The following did work:

• Manually launching fleet desktop with proper env vars: open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app
• Restarting the machine

Potential next steps:

• Provide an easy way for user/admin to manually launch Fleet Desktop if it does not come up
• Explore other ways we can launch Fleet Desktop, maybe like:

launchctl asuser $uid sudo -u "$currentUser" /usr/bin/open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app

• Reference: https://scriptingosx.com/2020/08/running-a-command-as-another-user/
• Example: https://github.com/bvanpeski/unActivationLock/pull/12/files#diff-04ce03e58be7bf3ed9bdaeb13cdce7c785148cb5142dc58811104563b0e97767

[getvictor]

@noahtalerman We need some product guidance regarding the next steps for this issue.

First, we don't know how big this issue is. Maybe the fix approach would be different if it were a rare issue instead of happening all the time.

I propose gathering some telemetry.

• fleetd would send the error and some details back to Fleet server
• Fleet server would store this in Redis for 7 days (or less)
• At this point, we can review this data for our cloud-hosted customers
• We can use this infrastructure for future/other errors as well

Regarding the issue itself, the thing that appears to work is resetting the host.

Options:

1. Force end user to restart the machine. Probably not desirable.
   • Even if we schedule the restart at night, end user may be annoyed the next morning.
2. Send the error back to Fleet server and mark the host with a warning like: "agent functionality is limited until the end user restarts their machine"
   • Besides not seeing Fleet Desktop, it is likely that some scripts would also fail if attempted to be run as user
3. Allow end user to launch Fleet Desktop manually.
   • The service desk would have to know about this and tell the end user how to do it.
   • We would dump out a script that the end user can run to launch Fleet Desktop (if fleetd failed to do so).

[noahtalerman]

Moved original bug description here for safekeeping:

💥  Actual behavior

I upgraded for macOS 14.5.

After my computer rebooted, the Fleet icon (Fleet Desktop) wasn't present in my menu bar.

🧑‍💻  Steps to reproduce

1. Install fleetd
2. Upgrade to macOS 14.5

🕯️ More info (optional)

Here are my fleetd logs:

• Fleet Desktop: https://drive.google.com/file/d/15Y-HXmzGG0wE1IhyZaaW0OPusWyYzo4v/view?usp=drive_link
• Orbit: https://drive.google.com/file/d/1issRUo1aqi2zFoKuFUUP5nC3JxJK_HeD/view?usp=drive_link

UPDATE: After restarting my computer, the Fleet icon appeared and I could access Fleet Desktop. I think it's worth time-boxing this one to see if there's anything reproducible in my logs (noahtalerman 2024-05-24)

[noahtalerman]

@getvictor thanks for the detailed summary of the debug session and proposed next steps!

I moved the summary of the debug session to the main issue description so other folks can see the latest status of the bug.

Gathering telemetry sounds like a good idea to me.

Re the solution/fix, is there a way we can allow the end user to launch Fleet Desktop that's not a script? Can we make it appear in macOS Spotlight Search? (CMD+space)

[getvictor]

@noahtalerman Yes, we can dump a config file, and allow the user to simply launch Desktop with: open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app

Putting it into Spotlight is a bit more difficult. We would have to move it to /Applications. Also, we would be risking end users deleting it by mistake.

[noahtalerman]

we can dump a config file, and allow the user to simply launch Desktop with: open /opt/orbit/bin/desktop/macos/stable/Fleet\ Desktop.app

@getvictor got it. I don't think we want the end user to have to open their Terminal to run a command.

Since the known solution that meets this^ requirement, Spotlight, is a relatively large effort and this bug doesn't occur often then I think we can close this bug w/ a documentation fix + adding telemetry.

I think we can document that if Fleet Desktop doesn't appear trying restarting the workstation.

Later we might come back to adding Fleet Desktop to Spotlight.

cc @lukeheath @dherder

[iansltx]

FWIW I ran across this issue on Sequoia after running Migration Assistant to try to repro #21317, on the source machine for Migration Assistant. Reinstalling Fleet Desktop didn't solve the issue but restarting the machine did. I don't seem to have a stable repro scenario though, as I didn't seem to have the issue when running the same Migration Assistant process a second time (only significant difference being manuall enrollment into MDM vs. no MDM, but I think that's a red herring).

[getvictor]

The effort to add telemetry is tracked by #23413

This ticket is in waiting status until we can analyze telemetry results from customers.

[lucasmrod]

A related issue (Fleet Desktop icon not showing): https://github.com/fleetdm/fleet/issues/23644. Related because some customers/users not seeing the icon could be caused by #23644 (and not this particular issue).

[getvictor]

@eashaw Can I get telemetry from PostreSQL for users running 4.59.0+ and/or instructions how to get it myself?

[getvictor]

@zayhanlon From our analytics, we see this error a lot for customer-preston. I put a note in their Slack.

For other customers, with ~180,000 hosts running fleetd on Fleet 4.59.1 and later, we see this issue 20 times in the last month or so.

So, this issue happens rarely, and we do not see it happening on macOS 15.

I recommend we wait another 2-4 weeks and take another sample of errors. If we do not see the issue on macOS 15, we can consider closing it without fixing it.