pkey: track whether pkey is private key or not

[rhenium]

There are multiple places where it's necessary to know whether a pkey is a private key, a public key, or just key parameters. Unfortunately, OpenSSL doesn't expose an API for this purpose (even though it has one for its internal use).

Currently, we drill down into the backing object, such as RSA, and see if the corresponding fields are set or not to determine it. This doesn't work on OpenSSL 3.0 because of the architecture changes.

Let's manually track this information in an instance variable for now. This has been partly done for ENGINE-backed pkeys. Now all pkeys get this flag.

PKeys are immutable on OpenSSL 3.0, so it just needs to be stored once on initialization. On OpenSSL 1.1 or before (including LibreSSL), it must be updated whenever a modification is made to the object.

This comes with a slight behavior change. PKey returned by following method will be explicitly marked as "public", even if it happens to point at an EVP_PKEY struct containing private key components. I expect the effect is minimum since these methods explicitly say "public key".

• OpenSSL::X509::Certificate#public_key
• OpenSSL::X509::Request#public_key
• OpenSSL::Netscape::SPKI#public_key

[no6v]

I recently found that OpenSSL::PKey::EC.new("prime256v1").to_der dumps core, and have been searching the way to check the pkey is public? or not briefly. This PR almostly implement that way, but the above code still dumps core. I'm not sure such instance makes any sense, but would you consider to fix this segv? Thanks.

[rhenium]

I recently found that OpenSSL::PKey::EC.new("prime256v1").to_der dumps core, and have been searching the way to check the pkey is public? or not briefly. This PR almostly implement that way, but the above code still dumps core. I'm not sure such instance makes any sense, but would you consider to fix this segv? Thanks.

ruby/openssl should not cause a segfault. This seems like a regression introduced by #328, which went into v3.0.

I hoped it would be checked by OpenSSL, like for all other EVP_PKEY types than EC, but it wasn't the case then.

[rhenium]

This is currently blocked by failing CI checks on Ubuntu 22.04. It appears to be a bug that has been fixed by OpenSSL 3.0.4: https://github.com/openssl/openssl/commit/9f3626f2473bdce53e85eba96e502e950e29e16f, specifically the diff in providers/implementations/encode_decode/decode_der2key.c. But Ubuntu's OpenSSL package is based on 3.0.2 and has yet to incorporate the patch.

I will need to figure out how I can request the bug fix backported, or come up with a workaround (if there is any).

[rhenium]

After some thought, this approach to fully track the state of a key may not work with the current OpenSSL 3.0.x API. We plan to provide a wrapper for EVP_PKEY_fromdata() to construct a key from a set of individual parameters (see #555) but it's seemingly not possible to tell whether the resulting key is a private key or a public key.

For now, I think we still have to continue to check against the low-level structure (and ignore the deprecation warnings, #576).