macOS MDM profiles marked as failed when re-enrolling during Fleet downtime

[jahzielv]

Fleet version: Dogfood rc-minor-fleet-v4.65.0

Web browser and operating system: macOS

---

💥  Actual behavior

Fleeties had to update our workstation MDM enrollment profiles today as part of scheduled maintenance. When I did this (removed enrollment profile, then went through the macOS native MDM enrollment modal), our dogfood Fleet server was down as a new RC was being deployed. After Fleet was back up, I expected the MDM profiles to be re-delivered without any interaction needed, but instead they were all marked as "Failed" in the Fleet UI.

🧑‍💻  Steps to reproduce

1. Remove Fleet MDM enrollment profile from an ADE host
2. Verify that all configuration profiles are removed from the host
3. Bring down the Fleet instance, then bring it back up
4. Refetch the host in the Fleet UI
5. Verify that the assigned configuration profiles are marked as "Failed", with this error message: "This setting had been verified by osquery, but has since been found missing on the host."

🕯️ More info (optional)

N/A

🛠️ To fix

@marko-lisica: Profiles should be delivered to a host once the enrollment profile is installed and the Fleet server is up.

[georgekarrv]

Hey team! Please add your planning poker estimate with Zenhub @getvictor @ghernandez345 @mna

[georgekarrv]

@noahtalerman @lukeheath kicking this one out of the sprint to accommodate the incoming P1.

[georgekarrv]

Had to kick this back out of the sprint to make room for https://github.com/fleetdm/fleet/issues/24475

[MagnusHJensen]

I was looking into reproducing this one, unfortunately it's not one I can tackle within the next 30 days, as when enrolling a device into ABM through the Apple Configurator app, there is a 30 day provisional period, where one can freely remove the enrollment profile and release the device from ABM, which I did and now my 30 days starts over, unfortunately.

So if one is over the 30 day provisional period then they can tackle it.

[mna]

Tried it first without bringing Fleet down:

• Removed enrollment profile via "System settings" -> "Profiles" -> "fleet enrollment" (click - and remove)
• In fleet, the host details showed as follows:

• Mac host went through the remote management wizard (automatically, without any user interaction on the host, no need to follow the steps in the UI message)
• Enrollment completed successfully
• Shortly after, custom settings (profiles) got deployed and status went to "Verifying"
• After a "Refetch" of the host, status went to "Verified"

[mna]

Now with Fleet server down when the enrollment profile is removed:

• Removed enrollment profile via "System settings" -> "Profiles" -> "fleet enrollment" (click - and remove)
• All profiles got removed on the host
• Remote management wizard did not come up
• Turn Fleet server back online
• Wait... still no Remote management Wizard
• Force a refetch, Host page updates to this:

• Remote management wizard showed up on host, completed successfully
• After a bit, custom profiles got installed, status went to "Verifying"
• After a forced refresh, status went to "Verified"

So with those steps, I could not reproduce the issue. My guess is that the bug is caused when the re-enroll (Remote management wizard) is done while Fleet is down, but for some reason it did not show up on my host until after I brought Fleet back up and forced a refresh. Might be a coincidence and I just need to wait longer, will try again.

[mna]

My guess is that the bug is caused when the re-enroll (Remote management wizard) is done while Fleet is down, but for some reason it did not show up on my host until after I brought Fleet back up and forced a refresh.

Actually that doesn't make sense, as Fleet needs to be up and running for the enrollment to work. Seems like I can't repro the bug, will try a few different things.

[mna]

Tried refreshing the host with Fleet back up and before going through the Remote management re-enroll wizard, results in this host page and triggers the wizard on the host:

Re-enroll completes successfully and custom profiles are installed shortly after. Status becomes "Verifying":

And triggering another host refresh transitions to "Verified":

[mna]

I see that in Jahziel's case, disk encryption was enabled, so I'll try again with that.

[mna]

Test with disk encryption turned on, state before unenrolling:

• Unenroll with Fleet offline
• Removed enrollment profile via "System settings" -> "Profiles" -> "fleet enrollment" (click - and remove)
• All profiles got removed on the host
• Turn Fleet server back online
• Force a refresh on the host page once the host shows back as online
• Triggers the re-enroll wizard, and refreshed page looks like this (before re-enrolling):

• Remote management wizard showed up on host, completed successfully
• After a bit, custom profiles got installed, status went to "Verifying" except disk encryption Pending (action required)

• After a forced refresh, status went to "Verified" except for disk encryption:

So I'm still unable to repro the bug. @georgekarrv thoughts on how to proceed, should I close it or add the "repro" label and ask for Gabe to take a shot at it too before we do so?

[mna]

Only small (unrelated?) issue I found is that the disk encryption message is still up after logging out and back in, so the action required is apparently misleading (even after a restart + refresh, message stays there, disk encryption remains pending):

[mna]

Gave it a shot with the exact version reported in this ticket (4.65.0), still wasn't able to repro. I'll add the "reproduce" label and ask @PezHub if he can take a shot at trying to repro too, and if he can't either, I think we should close this.

[PezHub]

Spent about an hour trying to repro without any luck. Closing the ticket.

[fleet-release]

In downtime's mist, Profiles once failed now sail, Secure, in Fleet's care.