🎸 Support macOS MDM "multi-tenancy"

[zwass]

Problem

Some customers would like to be able to support multiple separate connections to Apple (APNS cert + DEP token) in Fleet's MDM.

Scenarios:

1. Customer is a managed service provider and wants to be able to manage MDM on behalf of multiple separate organizations (that are their customers).
2. Customer is an organization that has separate business units (possibly through acquisitions) with separate ABM accounts.

Requirements

1. Each team can be configured with a separate APNS cert/key and DEP token.
2. New teams can be created and configured via API (this likely means solving the problem of storing secrets -- could be done through AWS Secrets Manager or similar, maybe we would also want an option to store them encrypted in the DB for non-cloud deployments?
3. Upon DEP enrollment, hosts automatically join the corresponding team.

Potential Solutions

@dherder drew up a diagram describing what this could look like

[marko-lisica]

@zayhanlon @zwass We didn't get to this one in the current design sprint. Adding it to feature fest.

[noahtalerman]

Hey @dherder, heads up, we didn't have room to take this one in the current design sprint (4.48).

[phtardif1]

We now also have a European MSP (confidential) that requires same set of features and willing to work with us on developing and bringing to market

[alexmitchelliii]

@noahtalerman prospect ibara would like to participate in design reviews with the team if that works for you.

[noahtalerman]

@alexmitchelliii sounds good. I added a reminder to the MDM design review doc to invite them to design review when we have wireframes for this story.

[noahtalerman]

Alex: Think about the user journey from the customer's perspective as someone who’s trying to automate provisioning a new customer of theirs.

cc @marko-lisica

[marko-lisica]

Hey @pintomi1989, we didn't design this one in the current sprint. We'll work on it next sprint.

[noahtalerman]

Updated this issue to user story format and moved the original issue description below for safekeeping.

---

Problem

Some customers would like to be able to manage multiple clients' workstations using one Fleet server:

• All clients use the same APNs cert/key.
• All clients use the same WSTEP cert/key.
  • Noah: This one could be pushed to a follow up pass.
• One client, of the customer, ~~can have many MDM servers (tokens) in ABM.~~ has one MDM server in ABM (Fleet).
  • ~~Noah: Why not one MDM server per client? So that the configuration profiles automatically applied to a single customer's Macs (upon enrollment) are different than the profiles applied to their iPhones.~~
    • Noah: I think we can simplify this for the IT admin so that one client has one MDM server in ABM. In Fleet, change from one ABM connection (supported today) to many ABM connections. One ABM connection can have a different default team for macOS, iOS, and iPadOS.
    • Noah: Each team in Fleet can have a different DEP profile. This means that one clients macOS hosts can get a different DEP profile than the same client's iOS hosts.
• One client has one Azure AD connection.
  • Noah: This one could be pushed to a follow up pass.
• One client has one VPP connection.
  • Noah: Assuming the MSP is the one w/ ABM credentials (logs in), why not one VPP connection for all clients? Guess (should confirm): technical limitation. If one host is associated w/ one ABM it can't install App Store apps added in another ABM.
  • Noah: One client has one ABM connection. Each ABM connection has one MDM server token, one VPP token, and different default team for macOS, iOS, and iPadOS. This way, the IT admin only sees and can only add App Store apps to the client's teams in Fleet.

Scenarios:

1. Customer is a managed service provider and wants to be able to manage workstations on behalf of multiple separate organizations (that are their customers).
2. Customer is an organization that has separate business units (possibly through acquisitions) with separate ABM accounts.

Requirements

1. Each team can be configured with a separate APNS cert/key and DEP token. UPDATE: Might not be necessary. We can use one APNs cert/key pair for many customers (noahtalerman 2024-07-03).

• Noah: The current plan is to use one APNs cert/key for many customers. Other MDM solutions are already doing.
• Noah: For DEP, the customer is setting up Apple Business Manager on behalf of their clients.

2. New teams can be created and configured via API (this likely means solving the problem of storing secrets -- could be done through AWS Secrets Manager or similar, maybe we would also want an option to store them encrypted in the DB for non-cloud deployments?
3. Upon DEP enrollment, hosts automatically join the corresponding team.
4. End user authentication workflows would have to unique SAML authentication streams per team.

• Noah: Not a requirement in the first pass. Something aspiration for later.

6. Admin SSO would also need to be configured per team.
7. Noah: The customer today has one VPP connection per client. Consider one VPP can be used for many teams. If we go w/ the approach of one ABM to one team then it would be lot easier to point many teams to the same VPP. Each client has one location.

Potential Solutions

@dherder drew up a diagram describing what this could look like

[georgekarrv]

Just adding for visibility. This ABM token needs to be associated on a per team basis for when we change the DEP profile (changing sso or setup assistant). So a mapping of team to abm token will be required.

[noahtalerman]

This ABM token needs to be associated on a per team basis for when we change the DEP profile (changing sso or setup assistant). So a mapping of team to abm token will be required.

@georgekarrv and I met today and I'm not convinced team <=> ABM association is the way to go.

We want unique automatic enrollment (DEP) profiles per team. To support host X being on team A and in ABM X while host Y on team A and in ABM Y, we might need a host <=> ABM association.

@roperzh heads up that I added you to design review tomorrow to discuss the above and get your eyes on the CLI/API changes.

DONE @noahtalerman:

• DONE: Expiration banners for ABM and VPP
• DONE: What happens when a team's name changes. Do we cascade this name change across these new ABM/VPP settings?
  • => Noah: Yes. Good thoughts from Rachel on options for how we do this here: https://github.com/fleetdm/fleet/pull/21043/files#r1704355956
• DONE: Error messages for CLI/API when the user tries to use the old way and the new way.

@noahtalerman questions:

• Can the UI use the new GET /apple_business_manager and GET /volume_purchasing_program endpoints instead of the apple_bm_terms_expired and apple_bm_enabled_and_configured booleans in GET /config? Deprecate these booleans.
• If an IT admin is using only VPP and no ABM, do we need to prompt them to accept new ABM terms?

[noahtalerman]

The TODOs in the above comment are now DONE.

When working on them, I realized we need API design for the upload/delete VPP endpoints and the upload/delete ABM endpoints.

VPP is in the open PR here (hasn't merged b/c we haven't shipped 4.55).

ABM is in the contributor docs here (upload) and here (delete).

We can break these endpoints because they're in the contributor docs.

Hey @georgekarrv can you please help w/ API design for those endpoints? Please commit on top of my PR here: https://github.com/fleetdm/fleet/pull/21043

[noahtalerman]

Hey @georgekarrv heads up that I added this TODO as a checkbox in the "Engineering" section:

• [ ] Contributor API endpoints to support best practice GitOps (fleetctl gitops) and backwards compatibility GitOps (fleetctl apply). Please add these to the existing reference docs PR https://github.com/fleetdm/fleet/pull/21043.

[noahtalerman]

Hey @PezHub heads up that I dropped this in the QA section of the issue so we don't forget:

• Be aware of a similar bug for this multiple ABM/VPP connections story: https://github.com/fleetdm/fleet/issues/21163
  • I think let's make sure to test that the team name change is reflected because we have made similar mistakes in the past.

[roperzh]

Engineering high-level game plan and proposal for database schema is being discussed at https://github.com/fleetdm/fleet/pull/21232/files

[nonpunctual]

[noahtalerman]

@nonpunctual thanks for dropping that!

Our goal is to have different Setup Assistant configurations for macOS / iPadOS that each correlate to a respective team

...auto-assign devices to a specific team (eg. Macs go to the Workstations team, iPads go to an iPad team)...

Heads up that, to achieve prospect-blondelet's goal, the best practice workflow will involve only one ABM connection (token).

As part of this story, we're adding the ability to tell Fleet to route Macs, iPhones, and iPads from the one ABM to a specific team in Fleet.

(this story also includes adding the ability to add multiple ABM connections)

For example, here's what this will look like for us at Fleet once we ship this story:

Only MSPs and organizations that have multiple ABMs through acquisitions (think Facebook acquiring Instagram) will want to add more than one ABM connection.

cc @dherder

[noahtalerman]

During MDM standup today we decided that, in this story, a team can only have one VPP token (v. a team having one or more VPP tokens).

Why? There are no known use cases / workflows in which a team (baseline) will need to pull App Store apps from many VPP connections. If we're wrong we can add support for this later.

Here's how this decision affects Fleet's interfaces (UI/API/CLI):

• UI: Edit teams modal here.
• API: PATCH /api/v1/fleet/vpp_tokens/:id/teams here
• CLI: GitOps here.

In addition to the above, Figma now includes wireframes to show the expected behavior right after the user adds a new ABM/VPP connection:

ABM here: macOS team, iOS team, and iPadOS team are set to "No team" by default.

VPP here: teams are set to empty by default.

Also, in addition to the above, Figma now includes wireframes here to show the expected behavior when a team has no VPP token and a team has no VPP apps (that haven't been added) here.

@roperzh and @ghernandez345, please let me know if you have any questions! Thanks :)

[noahtalerman]

Org name not unique on Apple side.

We want org name to be unique on the Fleet side. Why? UX decision. There are three best practices:

• Enterprise. One ABM token
• Managed Service Provider (MSP). Many ABMs and one token per ABM
• Enterprise that makes acquisitions. Many ABMs and one token per ABM

We can simplify the UX by guiding customers down one of these paths.

We’ve done some theoretical testing to see if this breaks and we haven’t found a real life scenario yet. That said, we want to keep the door open. We might learn that we’re wrong and remove this constraint later.

[marko-lisica]

We found out few bugs related to ABM and VPP workflows.

Currently in the UI we show to Fleet Free users card to connect VPP, which will be solved in this story, so I won't open dedicated bug for it.

There is another bug which I filed here: https://github.com/fleetdm/fleet/issues/21315

[gillespi314]

@noahtalerman @marko-lisica, where should the Fleet license expiration banner rank in this hierarchy (currently, it ranks just above the VPP banner but it isn't specified in the new Figma)?

[marko-lisica]

@gillespi314 @noahtalerman What happens if Fleet licence expires? Can users use premium features?

I think this one should go above ABM and VPP, below APNs expiration. The worst thing if ABM expires is that some hosts might fail enrollment until the token is renewed and it's premium feature. If a user can't use premium features after Fleet license expiration, they won't be able to renew the token anyway.

[noahtalerman]

What happens if Fleet license expires? Can users use premium features?

@marko-lisica we just show a banner. The user can continue to use premium features (they don't stop working).

Since that's the case, I think premium license expiration should be prioritized lower than APNs, ABM, and VPP, right? Because if those expire they do break features.

Marko, can you please make sure Figma is updated with what you decide? That way we can go back to Figma if we forget.

cc @gillespi314

[marko-lisica]

Since that's the case, I think premium license expiration should be prioritized lower than APNs, ABM, and VPP, right? Because if those expire they do break features.

I agree. I just updated Figma to reflect this.

cc @gillespi314

[noahtalerman]

@rachaelshaw I passed this one to you during confirm and celebrate. Can you please own closing the remaining checkboxes? Thanks :)

[noahtalerman]

Hey @rachaelshaw, I just noticed that the reference docs PR has conflicts. Please let me know how/if I can help resolve those. Happy to help!

Getting reference docs merged is last item before we can call this story done.

[noahtalerman]

Hey @rachaelshaw, I resolved the conflicts for the reference doc PR.

I left a question for Dante here but I think we can go ahead and merge if everything looks good. Then address any fixes to the redirect a follow up PR.

[noahtalerman]

Hey @rachaelshaw, just following up w/ another ping!

Please let me know if/how I can help get the reference doc PR across the finish line so we can call this story shipped.

[noahtalerman]

Updates to the permissions guide are in a PR here: Permissions PR is here: https://github.com/fleetdm/fleet/pull/22336

Waiting to closet this story until that is merged.

[noahtalerman]

Hey @zayhanlon and @dherder heads up that this story w/ customer/prospect labels attached was shipped in 4.56 🚀

We're missing the guide. I filed a bug for this here: https://github.com/fleetdm/fleet/issues/22339

I think let's leave this story open until we ship the guide.

[roperzh]

hey @noahtalerman the guide was already updated here https://github.com/fleetdm/fleet/pull/21627