Renew DigiCert certificates on macOS

[noahtalerman]

Goal

User story
As an IT admin,
I want to renew certificates from DigiCert certificate authority (CA) on my macOS hosts before they expire
so that I can be sure my end users can always access my organization’s Wi-Fi and VPN.

Key result

Deploy certificates from DigiCert and custom certificate authority (CA)

Original requests

• #13420

Context

• Product designer: @marko-lisica

Changes

Product

• [ ] Other changes:
  • Fleet already stores expiration date of certificates from DigiCert.
  • 30 days before expiration, Fleet should issue a new certificate from DigiCert and resend the configuration profile with new certificate.
    • If the certificate's validity period is less than 30 days, renew it half that number of days before expiration. For example, if the validity period is 20 days, renew it 10 days before expiration.
  • Fleet don't need to revoke certificate from DigiCert.
• [ ] UI changes: No changes.
• [ ] CLI (fleetctl) usage changes: No changes.
• [ ] YAML changes: No changes.
• [ ] REST API changes: No changes.
• [ ] Fleet's agent (fleetd) changes: No changes.
• [ ] GitOps mode changes: No changes.
• [ ] Activity changes: No changes.
• [ ] Permissions changes: No changes.
• [ ] Changes to paid features or tiers: Fleet Premium only. Covered in pricing table already.
• [ ] Transparency changes: No changes.
• [x] First draft of test plan added
• [ ] Other reference documentation changes: No changes
• [ ] Once shipped, requester has been notified
• [ ] Once shipped, dogfooding issue has been filed

Engineering

• [ ] Test plan is finalized
• [ ] Feature guide changes: #28077
• [ ] Database schema migrations: N/A
• [ ] Load testing: N/A

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

Requires load testing: No, because certificate renewals should be spread throughout the certificate lifetimes. Risk level: Low

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

• [ ] Make sure that configuration profiles that have DigiCert variables in it, are automatically resent when certificate is 30 days from expiration.
• [ ] Make sure that if you set certificate validity period in DigiCert certificate profile less than 30 days, Fleet should renew it half that number of days before expiration. For example, if the validity period is 20 days, renew it 10 days before expiration.
• [ ] Verify that certificates are added to a keychain
• [ ] Verify that certificates are displayed on host details page
• [ ] Verify that Fleet throws an error if the variable in CN changes but seat ID isn't changed.
• [ ] Make sure that when the host is moved to another team or DigiCert configuration profile is removed, certificate is also removed from keychain.

Happy path

1. Configure DigiCert CA in Fleet settings
2. Create DigiCert configuration profile. Use $FLEET_VAR_DIGICERT_PASSWORD for Password key, and $FLEET_VAR_DIGICERT_DATA for Data field.
3. Upload profile to Fleet and verify that it's delivered to a host (status verified on host details)
4. After the profile is verified, make sure that it's automatically resent 30 days (or half of the validity period) before expiration.
5. On the other host, manually resend the profile and verify that automatic renewal still works after that.

Testing notes

Confirmation

1. [ ] Engineer: Added comment to user story confirming successful completion of test plan.
2. [ ] QA: Added comment to user story confirming successful completion of test plan.

[lukeheath]

@noahtalerman @georgekarrv FYI this will need to be prioritized into the next sprint (4.68.0).

[noahtalerman]

FYI @marko-lisica ^

[lukeheath]

@georgekarrv This is still in drafting, but I'm adding a P2 because we'd like to work on this as top priority next sprint, so that we can deploy the feature to the customer's staging environment for review without waiting for the full release.

[JordanMontgomery]

Adding some notes based on the way this has been implemented:

Since digicert is still an experimental feature this will not currently renew digicert certificates requested prior to this renewal feature being added. Because of this you won't see certificates you might have requested when this was initially implemented start being renewed until you manually renew them (by resending the profile)

You can test this without waiting 30 days or however long by setting the not_valid_before and not_valid_after columns in the appropriate host_mdm_managed_certificates table row. not_valid_before is the date the certificate was issued and not_valid_after is the date the certificate expires. The validity window is not_valid_after - not_valid_before. The cleanups cron triggers this renewal task and it runs once per hour so there is a bit of waiting between the time a certificate becomes eligible for renewal(either naturally or by tweaking the DB) and the time the certificate actually gets renewed.

Also on macOS there is potentially a small window between when the new certificate gets sent down and when the old certificate is gone from the system so if you refetch quickly after sending a new certificate you might still see the old certificate information returned by OSQuery. This seems to be longer if you leave Keychain Access open, I assume because internally keychain access is keeping some kind of reference to the old cert valid longer than it otherwise would be, so close keychain access once you get done looking at what you're looking for for best results and in my testing once the old certificate disappears from keychain access it also no longer shows on a refetch. I don't think there is anything we can do to change this anyways.

[getvictor]

@JordanMontgomery FYI: osquery has a default 5-minute macOS keychain cache: https://osquery.readthedocs.io/en/stable/installation/cli-flags/#macos-keychain-flags

[JordanMontgomery]

https://osquery.readthedocs.io/en/stable/installation/cli-flags/#macos-keychain-flags

@getvictor thanks for pointing that out. I didn't even think of OSQuery caching, I just assumed it was something related to the behavior where I was seeing the certificates in the UI for a bit before they dropped off. I'll be sure to check OSQuery caching docs going forward

[PezHub]

QA Test Plan Results

✅ Successfully completed all tasks in the test plan including the Happy Path.

Example of expected error if the variable in CN changes but seat ID isn't changed -

Note: We have an outstanding question about expected behavior when a profile is resent, fails, but still shows as verified after refetch since the profile remained on the host. Slack thread

Once that is resolved, I will move to Ready for Release

[PezHub]

After discussing with the team we decided to file a bug for the above issue

[PezHub]

Bug has been resolved, moving to Ready for Release!

[fleet-release]

Certificates renew in time, Secure access, peace of mind. Fleet keeps systems prime.

[fleet-release]

Cert renewal swift, Ensuring secure access, Peace for IT admin.