Ability to add Fleet maintained apps to macOS setup experience in a GitOps-only environment

[eugkuo]

Goal

User story
As an IT admin,
I want to add FMAs to the macOS setup experience using GitOps
so that I can manage everything without using the UI.

Key result

Customer request

Original requests

• #30067

Context

• Product Designer: @noahtalerman
• Engineer: @ksykulev

Changes

Product

• [ ] UI changes: TODO
• [ ] CLI (fleetctl) usage changes: TODO
• [ ] YAML changes: TODO
• [ ] REST API changes: TODO
• [ ] Fleet's agent (fleetd) changes: TODO
• [ ] GitOps changes: TODO
• [ ] Activity changes: TODO
• [ ] Permissions changes: TODO
• [ ] Changes to paid features or tiers: TODO
• [ ] My device and fleetdm.com/better changes: TODO
• [ ] First draft of test plan added
• [ ] Other reference documentation changes: TODO
• [ ] Once shipped, requester has been notified
• [ ] Once shipped, dogfooding issue has been filed

Engineering

• [ ] Test plan is finalized
• [ ] Contributor API changes: TODO
• [ ] Feature guide changes: TODO
• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer: Added comment to user story confirming successful completion of test plan.
2. [ ] QA: Added comment to user story confirming successful completion of test plan.

[iansltx]

@ksykulev Per comments on the original request from @noahtalerman and @mostlikelee, we want to:

1. Add a setup_experience bool on the same level as self_service on custom packages, VPP apps, and FMAs, default false. Use that to determine whether software is included in setup experience.
2. Stub the macos_setup.software key in GitOps; if it exists in a config, hard-fail with a link to docs for the new way of specifying setup experience.

Plan will be that we don't have a deprecation period on this to keep the code clean and avoid drawing out instances where we have to support both cases. Throwing a prompt at an LLM or two to see if we can make a bot do the work of moving YAML around (we might want a step 3 above of providing a tool to move YAML around).

[iansltx]

Actually, scratch the LLM option for now, as the migration path is more nuanced. One nuance here is that if Team A and Team B both import a set of software, but Team A has a piece of software marked for setup experience and Team B does not, we now need two software files (one with the setup experience flag set, one without) where we previously had one.

@allenhouchins Does this tweak (and its downstream requirements) make sense? In our case the Workstations team has setup experience software defined, but the corresponding canary team doesn't, so for our own setup we'd need to split package YAMLs.

For reference, here's the prompt I threw into JetBrains Junie, as well as Zed + Claude Sonnet 4:

In `it-and-security`, for each app store app under `macos_setup.software`, remove it from there and add `setup_experience: true` under the corresponding entry in `app_store_apps` in the same team file. For each software package under `macos_setup.software`, remove it from there and add `setup_experience: true` inside the corresponding YAML file. Finally, remove the `macos_setup.software` key in the YAML file.

Note that the above does not cover the package YAML splitting nuance, and my attempts at making this work with an open-weights model small enough to run in 64GB unified memory failed.

[allenhouchins]

I much prefer the current set up of adding it to the software block to basically say "make this software available to this team" then creating a secondary reference in macos_setup.software to say "make this software available through this method". For any software we expect all employees to use ("critical"), I want it to be installed via macos_setup, auto-install if its removed, and available in self-service for patching and easy re-installation/troubleshooting. I don't want to have to create multiple software_title.yml to achieve this.

I'm already running into challenges with some of the recent changes we've made to software where I will have to create multiple software entries if I want software scoped differently in teams so anything we can do to handle additional logic behind the scenes and avoid having to create multiple software entries of the same software just to use it differently, will always get my vote.

run in 64GB unified memory failed.

64GB?! Yikes!

[noahtalerman]

I much prefer the current set up of adding it to the software block to basically say "make this software available to this team" then creating a secondary reference in macos_setup.software to say "make this software available through this method".

@allenhouchins interesting! I added you to Tuesday's #g-software design review to take a look at the current designs and discuss.

[noahtalerman]

Decided to push this story (T-shirt size: S) out of the current design sprint because we won't have space to ship it in 4.72. Here's what we're shipping instead:

• Fleet-maintained apps bugs
• https://github.com/fleetdm/fleet/issues/30240
• https://github.com/fleetdm/fleet/issues/27983

[iansltx]

@marko-lisica based on the late, significant scope reduction on #27983, there's now enough backend capacity in the 4.72 sprint to bring this in if it's ready in time, though based on the source of this original request the software GitOps update activity should be split into its own story, as the customer isn't asking for that, "core to Fleet" notwithstanding, as the activity work is likely the majority of the effort on this ticket if it's brought in. Splitting stories allows for easier parallelization, and allows for derisking both stories.

[iansltx]

@noahtalerman Based on linked issues, looks like we missed calling out some scope here. I'm assuming that allowing specifying macOS setup experience in fleet_maintained_apps, as well as in app_store_apps, is in scope here. Revising YAML docs so that's clearer now (as well as contributor API docs).

EDIT: Never mind. YAML docs don't include this explicitly, but it's dev note'd in a comment on the PR. And it's in the test plan.

[iansltx]

Turns out, the contributor API for this already has the correct (per-package) format, so these changes won't have a server-side component (all fleetctl).

[iansltx]

@noahtalerman I thought we had missed frontend scope here because there might have been changes to the modals/exports we added in #28110. But it looks like we don't show custom targets, self-service, and categories in the sample YAML there already so moving them around wouldn't make a difference to what we would display?

[noahtalerman]

it looks like we don't show custom targets, self-service, and categories in the sample YAML there already so moving them around wouldn't make a difference to what we would display?

@iansltx that's right!

[noahtalerman]

• [x] REST API changes: No changes

@iansltx also, for future stories can you please leave in the explicit "No changes" items in the "Product" section? (example above). I added them back this time.

This helps us remember during confirm and celebrate and 3 months from now whether or not we intended.

We've learned that removing the checkboxes introduces questions later like "did we mean to make UI changes for this story?"

[iansltx]

• [x] REST API changes: No changes

@iansltx also, for future stories can you please leave in the explicit "No changes" items in the "Product" section? (example above). I added them back this time.

This helps us remember during confirm and celebrate and 3 months from now whether or not we intended.

We've learned that removing the checkboxes introduces questions later like "did we mean to make UI changes for this story?"

@noahtalerman All of the "no changes" items are still in the issue description, under the "No-ops" heading ("Product no-ops" probably makes more sense here, and I've used that heading title on other tickets where I've done this). Sounds like you missed them because they were "below the fold" but they're there, and splitting them out should actually make the "we intentionally didn't do these things" rundown easier. Thoughts on maintaining that "split out into a subheading" pattern, but moving that subheading up above the Engineering section?

My reason for doing this tweak (was going to mention this in chat channels but ran out of time, but discussed this with @mostlikelee ahead of time) is that stories are generally 80-95% "No changes", and when spec'ing/dev'ing it's easy to miss actual work in a sea of "No changes" items, and missing a spec item while in spec or development has more serious consequences than having to scroll down in the Confirm and celebrate ritual.

Additionally, the act of explicitly moving items to a "these are the things that won't change" heading during spec provides another "are we sure this isn't changing" checkpoint prior to estimation. That actually wound up helpful on #30849, as some GitOps Mode UI changes had been miscategorized as generic UI changes. Moving the Figma link to the more appropriate checkbox decreased likelihood of confusion during both development (imminently) and Confirm and celebrate (later), while maintaining an explicit list of parts of the product that are out of scope for the story.

I hadn't PR'd a handbook/story template change for this because I wanted to dial in how it looked with real-world examples first, hence why you're seeing this for the first time here.

[noahtalerman]

stories are generally 80-95% "No changes", and when spec'ing/dev'ing it's easy to miss actual work in a sea of "No changes" items, and missing a spec item while in spec or development

@iansltx ah, I hear you but the current way has been working for the other product groups. We just have to look at all the checkboxes to see which have "No changes".

To keep things consistent across product groups and reduce change for now let's please keep the story issue template the same. I removed

Thanks for testing it out!

That actually wound up helpful on https://github.com/fleetdm/fleet/issues/30849, as some GitOps Mode UI changes had been miscategorized as generic UI changes.

It sounds like we (it was me) messed up in this case. That's a whoops from me. I forgot about the new GitOps mode checkbox.

[iansltx]

Pulled this back to Ready for spec as there are outstanding test plan questions that we're working through.

[mostlikelee]

@noahtalerman heads up that this story is getting pushed to sprint 46

[iansltx]

@jmwatts @marko-lisica @noahtalerman For the following test plan item:

In the team YAML, specify the old macos_setup.software key, leave it empty, and specify the new setup_experience for any software. Verify that you see this error messaged.

There's no straightforward way to get GitOps to behave this way; when deserializing, values in YAML always appear to be set/valud, even if the software key isn't provided. We might be able to reach into YAML and parse things manually to tell the difference between "key not set" and "key set and empty" but I'm thinking that the existence of an empty software key doesn't hurt anything in this particular case, so probably not worth it.

Of course, we can easily capture cases where the sotware list is not empty (and fail if we see setup experience set on both a package/app store definition and the setup experience software key), which matters most from a conflict resolution perspective I believe.

The above means that setting setup experience flags on individual packages with an empty-but-extant setup experience software key will result in a successful GitOps run with the package/app store app level flags winning out. Is that revision to the test plan aceptable?

On the bright side, I was able to differentiate between "setup experience flag isn't supplied" and "setup experience flag exists and is set to false", so we can fail fast when the setup experience software key has items in it but the setup experience flag is set on any software on the team (vs. only failing when setup experience is set to true somewhere).

[noahtalerman]

The above means that setting setup experience flags on individual packages with an empty-but-extant setup experience software key will result in a successful GitOps run with the package/app store app level flags winning out. Is that revision to the test plan aceptable?

@iansltx I think this is ok. I updated the test plan (below and in issue description).

FYI @jmwatts

• [ ] ~~In the team YAML, specify the old macos_setup.software key, leave it empty, and specify the new setup_experience for any software. Verify that you see this error messaged.~~
  • [ ] UPDATE: @noahtalerman: Tricky to build. Here's what we decided instead: setting setup_experience on individual packages with an empty macos_setup.software software key will result in a successful GitOps run with setup_experience flags applied to the relevant software.
• [ ] Repeat the step above with some software in the macos_setup.software key. You should see this error message.

I also updated the dev note with this decision: https://github.com/fleetdm/fleet/pull/30254/files#r2164318796

[lucasmrod]

@iansltx We are implementing https://github.com/fleetdm/fleet/issues/30877 for v4.74.0. Let me know when this is merged to main so we can add support for setting setup_experience: true on Linux packages.

[jmwatts]

QA Notes (In Progress)

• [x] Add App Store app in-line on the team YAML file and don't specify the new setup_experience key. Verify that the software isn't added to macOS Setup Experience when GitOps is run.

• [x] Add App Store app in-line on the team YAML file and specify the new setup_experience key, setting the value to true. Verify that the software is added to macOS Setup Experience when GitOps is run.

• [x] Add App Store app in-line on the team YAML file and specify the new setup_experience key, setting the value to false. Verify that the software is not added to macOS Setup Experience when GitOps is run.

• [x] Add Fleet-maintained app in-line on the team YAML file and don't specify the new setup_experience key. Verify that the software isn't added to macOS Setup Experience when GitOps is run.

• [x] Add Fleet-maintained app in-line on the team YAML file and specify the new setup_experience key, setting the value to true. Verify that the software is added to macOS Setup Experience when GitOps is run.

• [x] Add Fleet-maintained app in-line on the team YAML file and specify the new setup_experience key, setting the value to false. Verify that the software is not added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package in-line, on the team YAML file and don't specify the new setup_experience key. Verify that the software isn't added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package in-line, on the team YAML file and specify the new setup_experience key, setting the value to true. Verify that the software is added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package in-line, on the team YAML file and specify the new setup_experience key, setting the value to false. Verify that the software is not added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package YAML file and reference the package YAML on the team YAML. Don't specify the new setup_experience key. Verify that the software isn't added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package YAML file and reference the package YAML on the team YAML. Specify the new setup_experience key on the team YAML, setting the value to true. Verify that the software is added to macOS Setup Experience when GitOps is run.

• [x] Add a custom package YAML file and reference the package YAML on the team YAML. Specify the new setup_experience key on the team YAML, setting the value to false. Verify that the software is not added to macOS Setup Experience when GitOps is run.

• [x] ERROR PATH: Add a custom package YAML file and reference the package YAML on the team YAML. Specify the new setup_experience key within the package YAML, setting the value to true. Run GitOps and confirm there is an easy to understand error message.

Custom target (labels)

• [x] In the team YAML, set labels_include_any for a custom package that is defined in-line. Verify that the label inclusions are applied when GitOps is run.

• [x] In the team YAML, set labels_include_any for a custom package that is referenced by path. Verify that the label inclusions are applied when GitOps is run.

• [x] In the team YAML, set labels_exclude_any for a custom package that is defined in-line. Verify that the label exclusions are applied when GitOps is run.

• [x] In the team YAML, set labels_exclude_any for a custom package that is referenced by path. Verify that the label exclusions are applied when GitOps is run.

• [x] In the package YAML, set labels_include_any to include a label that exists in Fleet. Verify that the key no longer works and you see an easy to understand error message when GitOps is run.

• [x] In the package YAML, set labels_exclude_any to exclude a label that exists in Fleet. Verify that the key no longer works and you see an easy to understand error message when GitOps is run.

Categories

• [x] In the team YAML, set categories to include one or more categories that exist in Fleet for a custom package that is defined in-line. Verify that the categories are applied when GitOps is run.

• [x] In the package YAML, set categories to include one or more categories that exist in Fleet. Verify that the key no longer works and you see an easy to understand error message when GitOps is run.

No team

• [x] Verify that setup experience, custom targets (labels), and categories work as defined above for "No team" (no-team.yml)

Backwards compatibility

• [x] In the team YAML, specify the old macos_setup.software key, add software in-line, and do not specify the new setup_experience for any software. Confirm the software is added to setup experience.

• [x] setting setup_experience on individual packages with an empty macos_setup.software software key will result in a successful GitOps run with setup_experience flags applied to the relevant software.

• [x] Repeat the step above with some software in the macos_setup.software key. You should see this error message.

GitOps speed

• [🔴 ] Run GitOps with 99 packages and 500 teams before and after this change. Verify that the speed of GitOps isn't negatively impacted. Prior to migration:

./gitops.sh 17.68s user 6.29s system 1% cpu 28:14.74 total

Post migration, no other changes:

./gitops.sh  23.06s user 8.57s system 1% cpu 45:00.91 total

Migration script

• [x] Run the script to migrate the the old package YAML format to the new format and verify that the script works. https://github.com/fleetdm/fleet/issues/31165

Generate-GitOps

• [x] fleetctl generate-gitops includes the setup experience flag on software for which it is set.

[jmwatts]

While testing this, also found #32997.

[jmwatts]

@marko-lisica @noahtalerman There seems to be an increase in the amount of time for the gitops runs.

With 97 packages and 300 teams. starting with a GitOps run that was successful on 4.73.1 then migrating the files and running against 4.74 I noticed a difference in time it took for the GitOps run to complete.

Prior to migration it took about 28 minutes:

sh ./gitops.sh 17.68s user 6.29s system 1% cpu 28:14.74 total

Post migration, it took about 45 minutes:

./gitops.sh  23.06s user 8.57s system 1% cpu 45:00.91 total

I am going to revert to 4.73.1 again and rerun the baseline, then will try after the migration again to see if these results are consistent.

I don't have a threshold right now to say whether or not this change is acceptable. Can you let me know what time delta is acceptable in this case?

[jmwatts]

Re-ran a few times per version:

4.73.1 ./gitops.sh 17.77s user 5.80s system 0% cpu 53:55.61 total ./gitops.sh 20.91s user 8.01s system 0% cpu 52:42.73 total ./gitops.sh 20.90s user 8.38s system 1% cpu 44:45.26 total

4.74.0 ./gitops.sh 25.35s user 9.63s system 1% cpu 45:31.24 total ./gitops.sh 23.50s user 10.48s system 0% cpu 58:26.00 total ./gitops.sh 22.02s user 10.06s system 0% cpu 53:46.92 total

it looks like the timing varies, but doesn't seem to have as drastic a change as I saw before, so I'm comfortable moving this one to "Ready for release"

[fleet-release]

In the cloud city, YAML shapes team's software scope, Fleet sails on smooth seas.