macOS Security Framework fails to import modern PKCS#12 created by OpenSSL 3

[danni-m]

Hi,

Im having the following error when running on macbook (M1) with a k3s cluster that was created by k3d:

 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/test_kube`
Error: SslError: MAC verification failed during PKCS12 import (wrong password?)

This doesn't happen if I use GKE or token based authentication.

I have a repository that reproduces this on my machine: https://github.com/danni-m/PKCS12_issue. The kubeconfig file im using is:

---
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTXpVNU1qa3pNRGd3SGhjTk1qRXhNVEF6TURnME9ESTRXaGNOTXpFeE1UQXhNRGcwT0RJNApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTXpVNU1qa3pNRGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUblVCS1NsT2FpN1J6UTVEandKQktaZFNFYThSTWsvRXpONEJ4N1pnSkQKNjlFS2xRZXZERkl3Mm1rMVpSNzNFNytxR2VpRVlLLzhadW5Tb0tCNGtwdjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUU0bjlOUzBUODdWaWx2d0hnZ2dSCjU1R1RhWXN3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnVlhBRUxBZW5IbVRhVU1GTjViaWZzaE9qYTN3VmRRVm4KTTRXZnhNT0VHWWtDSUdRRE1JT1Vtb2xtd2dEVmZabXo2bGt0Y2lxUHBtRHVxenNrZG9YZ0hiVXUKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://0.0.0.0:58362
  name: k3d-testing
contexts:
- context:
    cluster: k3d-testing
    user: admin@k3d-testing
  name: k3d-testing
current-context: k3d-testing
kind: Config
preferences: {}
users:
- name: admin@k3d-testing
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJT01iMWYvOHk4a293Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOak0xT1RJNU16QTRNQjRYRFRJeE1URXdNekE0TkRneU9Gb1hEVEl5TVRFdwpNekE0TkRneU9Gb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJQbHMxY2ZYdGpQVWIxdUMKZFhCbHVydTBZU3pqR0pGWlEzYXRUTHFDT1FTNlFDQUNMcW5NY29scy82aGhBL2RXY3JJdmFFS1VpWHZIMUs5dApzQkw5OEZxalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCU0VzRDdNZElVQnExNGJWdFoybjJ0S1pOMnY4REFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlBRzB2Yjk4dzV4ekVIL2tORTNBOGh1TmMwRG42N08yMS9WUmtzbFloSWN6Z0loQU9SZlVnSVpaWm40WU54egoydGpUMkhHdUlXS0QvVzVRdGp3Uk5pZjFWWEZjCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyTXpVNU1qa3pNRGd3SGhjTk1qRXhNVEF6TURnME9ESTRXaGNOTXpFeE1UQXhNRGcwT0RJNApXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyTXpVNU1qa3pNRGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUMWVDVlhyeTc4V3RSaS9iekQ4c1pZbkIwOUVTZDJCK0lHdDR6c0tTdXMKbWJERGZOWTdvSjhwWGJTeUpyNWkvdCs5VXBoTWljbGVYcHZFUjQycHhLaEZvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWhMQSt6SFNGQWF0ZUcxYldkcDlyClNtVGRyL0F3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQU94bmIrSTZYWEdFNFJ0RWpnelhacG9nOVZxMWJ0cGgKSklWcTRNbllseWJvQWlCNUQ3anFudjRnZGUrRFJFLzRtQ2Uxdk16d1JUSm9ZbnJTcUx3a2VNeGpodz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUtiU3B6N0NXdFNLZ3FaUHhHWm9tZTZCa1Z6RGxEbkxCRjF4MzFMZEh5dDBvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFK1d6Vng5ZTJNOVJ2VzRKMWNHVzZ1N1JoTE9NWWtWbERkcTFNdW9JNUJMcEFJQUl1cWN4eQppV3ovcUdFRDkxWnlzaTlvUXBTSmU4ZlVyMjJ3RXYzd1dnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

[clux]

Because you are using rustls, this looks like a duplicate of #542. I'd suggest trying native-tls for now.

[danni-m]

@clux Its also happens with native-tls

[danni-m]

@clux I pushed the change to my repo to reflect how i use native-tls.

[clux]

Hm. Damn. It runs successfully on linux against my k3d with rustls-tls (EDIT: and native-tls).

[danni-m]

A workaround for this will be to add openssl-sys = "=0.9.66" to your dependencies to avoid the implicit upgrade to openssl3.

[clux]

As noted on discord, this is a regression from kube 0.58.0, but only because 0.63 pulls in the new patch release of openssl-sys that pulls in the openssl 3 from their release on halloween: https://github.com/sfackler/rust-openssl/blob/master/openssl-sys/CHANGELOG.md#v0969---2021-10-31 (apparently openssl major version changes is not consider breaking for them).

It would be good to have some more eyes on this particularly upstream to find the underlying issue. Pinning openssl-sys is probably only liable to work for so long.

~~It's strange that it appears on rustls variant of that test case, but the test case is also not disabling default features so both dependencies will be present.~~

[clux]

Purging openssl from the dependency tree:

[dependencies]
log = "0.4.14"
anyhow = "1.0.44"
kube = { version = "0.63.2", default-features=false, features = ["derive", "rustls-tls", "client"]}
tokio = { version = "1.0.1", features = ["rt-multi-thread", "time", "fs", "macros", "net"] }
k8s-openapi = { version = "0.13.1", features = ["v1_20"], default-features=false }

gives the error

Error: SslError: No valid private key was found

from https://github.com/kube-rs/kube-rs/blob/120d0001ce58eb18ac59d2a9da1bf8514bc8058c/kube-client/src/client/tls.rs#L90-L117 - so that's the rustls EC issue again.

So it looks like this issue here is related solely to openssl 3.0 (additionally) breaking it on native-tls (and only on mac).

[clux]

This is a problem because that means that soon there could be no good way to run kube against k3d on mac's since both tls stacks have issues (at least if the pinning of openssl-sys stops being available). If anyone is able to dig in here and look for some workarounds it would be really appreciated.

[kazk]

As noted on discord, this is a regression from kube 0.58.0, but only because 0.63 pulls in the new patch release of openssl-sys that pulls in the openssl 3 from their release on halloween: sfackler/rust-openssl@master/openssl-sys/CHANGELOG.md#v0969---2021-10-31 (apparently openssl major version changes is not consider breaking for them).

• Upgraded the vendored OpenSSL to 3.0.0.

I don't think we're using vendored feature. Actually, enabling vendored feature might be a possible workaround too (avoiding openssl-sys).

A workaround for this will be to add openssl-sys = "=0.9.66" to your dependencies to avoid the implicit upgrade to openssl3.

The openssl-sys crate will automatically detect OpenSSL installations via Homebrew on macOS and vcpkg on Windows. Additionally, it will use pkg-config on Unix-like systems to find the system installation.

openssl - Rust

v0.9.67 added support for LibreSSL 3.4.0.

@danni-m What's the output of openssl version?

[kazk]

I don't think we're using vendored feature.

Never mind, I just found @danni-m's repo that enables it.

[danni-m]

@kazk

> openssl version
LibreSSL 2.8.3

[kazk]

Sorry, I didn't realize you had vendered enabled when I commented. openssl-sys simply builds openssl-src when vendored, so that shouldn't be relevant.

I'm currently reading https://www.openssl.org/blog/blog/2021/09/07/OpenSSL3.Final/ and https://www.openssl.org/docs/man3.0/man7/migration_guide.html to see if there's any breaking changes that might be relevant to us. I'm guessing using " " as a password is the cause:

https://github.com/kube-rs/kube-rs/blob/7cbd8cde88d713f5b21e1d409ac7064fab64b49f/kube-client/src/client/tls.rs#L7

https://github.com/kube-rs/kube-rs/blob/7cbd8cde88d713f5b21e1d409ac7064fab64b49f/kube-client/src/client/tls.rs#L17-L21

[kazk]

I confirmed https://github.com/danni-m/PKCS12_issue works fine on Linux as well.

I think the problem is pkcs12_from_pem using OpenSSL in all target_os:

https://github.com/kube-rs/kube-rs/blob/7cbd8cde88d713f5b21e1d409ac7064fab64b49f/kube-client/src/client/tls.rs#L40-L48

But native_tls::Identity::from_pkcs12 doesn't use OpenSSL on macOS and Windows.

https://github.com/kube-rs/kube-rs/blob/7cbd8cde88d713f5b21e1d409ac7064fab64b49f/kube-client/src/client/tls.rs#L19-L20

https://github.com/sfackler/rust-native-tls/blob/41522daa6f6e76182c3118a7f9c23f6949e6d59f/src/lib.rs#L176-L179

    pub fn from_pkcs12(der: &[u8], password: &str) -> Result<Identity> {
        let identity = imp::Identity::from_pkcs12(der, password)?;
        Ok(Identity(identity))
    }

https://github.com/sfackler/rust-native-tls/blob/41522daa6f6e76182c3118a7f9c23f6949e6d59f/src/lib.rs#L113-L121

#[cfg(any(target_os = "macos", target_os = "ios"))]
#[path = "imp/security_framework.rs"]
mod imp;
#[cfg(target_os = "windows")]
#[path = "imp/schannel.rs"]
mod imp;
#[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "ios")))]
#[path = "imp/openssl.rs"]
mod imp;

Some change in OpenSSL 3 is incompatible with them.

[kazk]

Probably this:

PKCS#12 API updates

The default algorithms for pkcs12 creation with the PKCS12_create() function were changed to more modern PBKDF2 and AES based algorithms. The default MAC iteration count was changed to PKCS12_DEFAULT_ITER to make it equal with the password-based encryption iteration count. The default digest algorithm for the MAC computation was changed to SHA-256. The pkcs12 application now supports -legacy option that restores the previous default algorithms to support interoperability with legacy systems.

https://www.openssl.org/docs/man3.0/man7/migration_guide.html#PKCS-12-API-updates

[kazk]

Yeah, found MacOS security framework fails to import RFC 7292 compliant PKCS #12 v1.1 file into keychain using modern cyphers.

It should be noted that OpenSSL 3 will by default move to AES-256-CBC for the encryption by default for the certificate and private key PBE algorithm instead of the legacy RC2-40 or 3DES.

[kazk]

We should be able to work around this by configuring Pkcs12::builder with legacy params when the target OS is macos/ios.

https://github.com/sfackler/rust-openssl/blob/15f263eb7eb61bc241ddc8bc0e5d79b68cb37936/openssl/src/pkcs12.rs#L74-L92

    /// Creates a new builder for a protected pkcs12 certificate.
    ///
    /// This uses the defaults from the OpenSSL library:
    ///
    /// * `nid_key` - `nid::PBE_WITHSHA1AND3_KEY_TRIPLEDES_CBC`
    /// * `nid_cert` - `nid::PBE_WITHSHA1AND40BITRC2_CBC`
    /// * `iter` - `2048`
    /// * `mac_iter` - `2048`
    pub fn builder() -> Pkcs12Builder {
        ffi::init();

        Pkcs12Builder {
            nid_key: Nid::UNDEF,  //nid::PBE_WITHSHA1AND3_KEY_TRIPLEDES_CBC,
            nid_cert: Nid::UNDEF, //nid::PBE_WITHSHA1AND40BITRC2_CBC,
            iter: ffi::PKCS12_DEFAULT_ITER,
            mac_iter: ffi::PKCS12_DEFAULT_ITER,
            ca: None,
        }
    }

Setting the parameters to the commented ones should work. The documentation is outdated with OpenSSL 3.

[kazk]

I'm hoping PKCS#8 support is merged soon (https://github.com/sfackler/rust-native-tls/pull/209). Once we have that, we don't need to depend on openssl on macOS and Windows anymore.

[kazk]

@danni-m If #693 still doesn't work after fcf3a9e (#693), we added openssl-tls feature that uses openssl for TLS on all platforms. You can try that by using the master branch.

[timdesi]

Hi All,

I may confirm that latest master have same problem. Testing with macOS Monterey.

Running /kube-rs/target/debug/examples/job_api
Error: native tls error: failed to deserialize DER-encoded PKCS #12 archive: MAC verification failed during PKCS12 import (wrong password?)

Caused by:
    0: failed to deserialize DER-encoded PKCS #12 archive: MAC verification failed during PKCS12 import (wrong password?)
    1: MAC verification failed during PKCS12 import (wrong password?)

[kazk]

Yeah, it's not fixed.

I'd recommend trying openssl-tls feature instead. native-tls feature exists for a historical reason (kube used to use reqwest), and it doesn't make much sense because macos/windows still depends on openssl anyway for creating PKCS#12.

PKCS#8 support in rust-native-tls was merged a few days ago, but I realized we still need to depend on openssl because we need to support other private key formats. We might be able to work around this issue with macos by converting to PKCS#8 instead though.

[timdesi]

Hi, i could confirm that now works.

Thx. for your great work.

❯ cargo run --example event_watcher

    Finished dev [unoptimized + debuginfo] target(s) in 0.26s
     Running `/kube-rs/target/debug/examples/event_watcher`
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://192.168.64.2:8443/api/v1/events? otel.name="list" otel.kind="client"
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] requesting
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] HTTP; http.status_code=200
[2022-03-31T13:22:36Z INFO  event_watcher] New Event: Created container dnsutils (via Pod dnsutils)
[2022-03-31T13:22:36Z INFO  event_watcher] New Event: Started container dnsutils (via Pod dnsutils)
[2022-03-31T13:22:36Z INFO  event_watcher] New Event: Container image "k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3" already present on machine (via Pod dnsutils)
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://192.168.64.2:8443/api/v1/events?&watch=true&resourceVersion=438543&timeoutSeconds=290&allowWatchBookmarks=true otel.name="watch" otel.kind="client"
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] requesting
[2022-03-31T13:22:36Z DEBUG kube_client::client::builder] HTTP; http.status_code=200