InstallEnterpriseApplication MDM command running on unintended Hosts

[nonpunctual]

Fleet version: Fleet 4.64.2

Browser verson: Latest Chrome

💥  Actual behavior

An InstallEnterpriseApplication mdm command is installing a bootstrap package which includes Munki on Hosts not in scope from customer-eponym Fleet instance. The bootstrap package is only intended to be installed by customer-eponym on Hosts that are enrolling for the 1st time.

🧑‍💻  Steps to reproduce

Unclear how to reproduce.

Theory for reproduction: any Host that has its enrollment profile replaced or renewed may be seen as a net-new, OOB enrollment.

• Configure bootstrap package to install on new enrollment
• Renew / replace enrollment profile on Host already enrolled in Fleet
• issue would be reproduced if mdm InstallEnterpriseApplication command is queued up for the Host on which enrollment profile was replaced

Related code:

https://github.com/fleetdm/fleet/blob/f3835e6d49283427a89edf4fca387113b5c282a9/server/worker/apple_mdm.go#L357

https://github.com/fleetdm/fleet/blob/6b7d23252272f8982fbae4948be0b500501bfa15/server/worker/apple_mdm.go#L123

🕯️ More info (optional)

• customer-eponym does NOT believe that any other automation solution in the environment is causing this issue.
• A recent CS action was taken on customer-eponym Fleet instance to remove unscoped profiles on Mar 12 2025.
• A curl command was run against customer-eponym Fleet instance to collect pending InstallEnterpriseApplication mdm commands like:

GET /api/v1/fleet/commands?request_type=InstallEnterpriseApplication&order_key=status&order_direction=desc

Full output from this command is posted in the customer Slack channel.

• customer-eponym reported that there are many duplicate Hosts in this pending mdm command output, e.g.,

      "host_uuid": "...ADDE55C5A122",
      "command_uuid": "18d82606-194e-482c-9220-0cc3f63dd64c",
      "updated_at": "2025-03-15T03:19:59.193891Z",
      "request_type": "InstallEnterpriseApplication",
      "status": "Pending",
      "hostname": "blah.home"
    },
    {
      "host_uuid": "...ADDE55C5A122",
      "command_uuid": "407447a9-1376-4ef7-9d3b-9fd0db581373",
      "updated_at": "2025-03-15T00:19:59.168679Z",
      "request_type": "InstallEnterpriseApplication",
      "status": "Pending",
      "hostname": "blah.home"
    },
    {
      "host_uuid": "...ADDE55C5A122",
      "command_uuid": "59c21c26-81d4-4659-bb27-b9c64ba3cb5a",
      "updated_at": "2025-03-15T01:19:59.198483Z",
      "request_type": "InstallEnterpriseApplication",
      "status": "Pending",
      "hostname": "blah.home"
    },
    {
      "host_uuid": "...ADDE55C5A122",
      "command_uuid": "8eaa3b30-7a38-418d-8518-8b7cf1c62238",
      "updated_at": "2025-03-15T00:19:59.242586Z",
      "request_type": "InstallEnterpriseApplication",
      "status": "Pending",
      "hostname": "blah.home"
    },
    { 

...

[zayhanlon]

@lukeheath i brought this to the #g-mdm channel and Noah suggested he thinks it's a p2. can you review?

Brock and Kathy are actively working with the customer and we can help either QA or an Engineer work with them to show reproduction or screenshare some things (however we aren't able to reproduce this workflow)

[lukeheath]

@zayhanlon Agreed! @georgekarrv I'm bringing this straight to the release board.

[georgekarrv]

@lukeheath that is more than we normally do with a p2

[lukeheath]

@georgekarrv Any high-priority issue is eligible for escalation into the current sprint based on impact. Because this is impacting software delivery in production, we need to dig in before next sprint.

[gillespi314]

@nonpunctual, this should be updated to refer to InstallEnterpriseApplication instead of InstallApplication, right?

[nonpunctual]

@gillespi314 Yes I will change. Sorry that was a lot of copy + paste....

[georgekarrv]

@gillespi314 let us know if there is anything else you need and any updates from the call this morning

[PezHub]

QA Test Results

Test Plan -

1. Upload Bootstrap pkg to a team
2. Enroll host via ADE
3. In the DB under nano_cert_auth_associations find the host UUID
4. Adjust the cert to be not valid after ~10 days from today. Eg. 4-12-25
5. Trigger cleanups cron job
6. Fleet should send a new install profile command with the fleet enroll profile
7. This triggers the host to install fleetd, profiles, and bootstrap pkg

Results -

Before fix = Bootstrap pkg gets reinstalled After fix = Bootstrap pkg does not get reinstalled

Additional Testing -

1. Run sudo profiles renew -type enrollment on the host - Before fix = Profiles get reinstalled and Bootstrap pkg gets reinstalled After fix = Profiles get reinstalled ~~but~~ and the Bootstrap pkg ~~does not get reinstalled~~ gets reinstalled.
2. Silent migrations (turn mdm off/on)? Before fix = Profiles get reinstalled and so does Bootstrap pkg After fix = Profiles get reinstalled , and so does Bootstrap pkg *It would be good to document this

EDIT: After revisiting these scenarios for a different ticket I realize I made an error and have corrected it above: The bootstrap pkg does get re-installed when running sudo profiles renew -type enrollment

[fleet-release]

Unintended hosts bloom, Install command finds its room, Fleet, in cloud, refines.