macOS 14 jobs hanging on security remove-trusted-cert command

[djc]

Description

I've written up the issue in some detail in https://github.com/rustls/rustls-native-certs/issues/179.

Repeating some salient bits here, this script (executed with sudo) ends up hanging:

+ ANY_CA_PEM=integration-tests/one-existing-ca.pem
+ ANY_CA_SUBJECT='OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ security authorizationdb write com.apple.trust-settings.admin allow
YES (0)
+ reset
+ security remove-trusted-cert -d integration-tests/one-existing-ca.pem
SecTrustSettingsRemoveTrustSettings: The specified item could not be found in the keychain.
+ true
+ list
+ cargo test util_list_certs -- --nocapture
+ grep 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
cert[95] = OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign
+ test_distrust_existing_root
+ assert_exists 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ list
+ cargo test util_list_certs -- --nocapture
+ grep 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ security add-trusted-cert -d -r deny integration-tests/one-existing-ca.pem
+ assert_missing 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ set +e
+ list
+ cargo test util_list_certs -- --nocapture
+ grep 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ ret=1
+ set -e
+ test 1 -eq 1
+ reset
+ security remove-trusted-cert -d integration-tests/one-existing-ca.pem
[ .. hangs .. ]

It seems to have worked on macOS 14.7.4 runners (image 20250331.1204) and failed with macOS 14.7.5 (image 20250421.1374). I since also noticed https://github.com/actions/runner-images/issues/11893 which suggests that our script might stop working in macOS 15, but since this is still on macOS 14.7 I'm not sure if that is related to the current failure.

Successful run, failed run.

(Previously discussed in #4734.)

Platforms affected

• [ ] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 22.04
• [ ] Ubuntu 24.04
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [x] macOS 14
• [ ] macOS 14 Arm64
• [ ] macOS 15
• [ ] macOS 15 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022
• [ ] Windows Server 2025

Image version and build link

20250421.1374

Is it regression?

yes

Expected behavior

Don't hang.

Actual behavior

Hang.

Repro steps

See above.

[aartis17]

Hi @djc, We will look into the issue and keep you posted with the updates. Thank you.

[sureshe456]

Hi All,We are currently investigating the issue and working on resolving it. We will keep you updated with any progress. Thank you.

[pronebird]

I have started having issues running security add-trusted-cert .. without sudo. Currently macos-latest runs macOS 14.7.5. Adding sudo solved the problem.

[djc]

To be clear, our project is already using sudo (but one level up, in executing the script).

[sureshe456]

Hi @djc, We are currently waiting the vm rollout of new macOS 14.7.6 runner. The updated version is already available in the macos-latest image level, but it's deployment is scheduled for next week due to MSbuild deployment freeze this week. We appreciate your patience and will notify to you as soon as deployment is complete. Hopefully, your issue will be resolved once macOS 14.7.6 available.

[sureshe456]

Hi @djc We could see that script is still failing even macOS 14.7.6 is available. We are investigating the issue in dig deeply and will keep you posted any further updates.Thanks.

[sureshe456]

Hi @djc Maybe something has been changed in macOS 14.7.5 due to that script is not working properly. We are still investing the issue and will keep you posted on any updates. Thank you for your patience.

[sureshe456]

Hi @djc After investigation,It appears that the command security remove-trusted-cert -d integration-tests/one-existing-ca.pem can't find the certificate in the system keychain, which is causing the script to fail. We have tested the following workaround to remove the certificate on macOS 14 runner, and it works as expected.

So, Please use the below workaround to resolve the issue and let us know if you have any further updates.Thanks.

- name: Manually remove trusted certificate
  run: |
    CERT_HASH=$(openssl x509 -in integration-tests/one-existing-ca.pem -noout -fingerprint -sha1 | cut -d= -f2 | tr -d ':')
    sudo security delete-certificate -Z "$CERT_HASH" /Library/Keychains/System.keychain

[djc]

I tried this in https://github.com/rustls/rustls-native-certs/pull/190, but it doesn't appear to work.

[sureshe456]

Thank you for your update. We will look into the another workaround to mitigate the issue.

[sureshe456]

Hi @djc , We have tested another workaround on macOS 14 images, it is working fine in my local. So. Kindly use the below code on your end and test it, If everything is fine then please update me. Thanks.

#!/bin/bash

set -ex

ANY_CA_PEM=integration-tests/one-existing-ca.pem
ANY_CA_SUBJECT="OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign"

reset() { 
  CERT_HASH=$(openssl x509 -in $ANY_CA_PEM -noout -fingerprint -sha1 | cut -d= -f2 | tr -d :)
  security delete-certificate -Z "$CERT_HASH" /Library/Keychains/System.keychain || true
  list | grep "$ANY_CA_SUBJECT" || true
}

list() {
  cargo test util_list_certs -- --nocapture 2>/dev/null
}

assert_missing() {
  set +e
  list | grep "$1"
  ret=$?
  set -e
  test $ret -eq 1
}

assert_exists() {
  list | grep "$1" > /dev/null
}

test_distrust_existing_root() {
  assert_exists "$ANY_CA_SUBJECT"
  security add-trusted-cert -d -r deny $ANY_CA_PEM
  assert_missing "$ANY_CA_SUBJECT"
  reset
}

security authorizationdb write com.apple.trust-settings.admin allow

reset
test_distrust_existing_root
printf "\n*** All tests passed ***\n"

[djc]

This appears to work for me, thanks!

[sureshe456]

Hi @djc Kindly provide your confirmation to close the ticket. Thanks.

[djc]

It still doesn't seem to work -- see https://github.com/rustls/rustls-native-certs/pull/190#discussion_r2200675580.

[sureshe456]

Hi @djc ,Lets check the any further workaround. Will keep you posted on any updates.

[sureshe456]

Hi @djc, Based on our analysis, the assert function is behaving unexpectedly—the test fails when the certificate is derivable. According to this comment on issue #12116, there does not appear to be a certificate issue on the macOS 14 runner image. Could you please review the implementation of the assert function to ensure it is working as intended?

[sureshe456]

Hi @djc Since there hasn’t been any further activity or follow-up, we’re closing this issue. If the problem persists on macOS 14 image, please feel free to reopen or create a new issue. Thanks!