Support for externally managed asymmetric keys

[reneme]

In the context of the KBLS project (#2500, website) we are looking into supporting private key material managed by the operating system. E.g. stored in the OS keychain or a secure element like TPM, Apple T2 and the like.

Eventually, we'd hope to have a platform-independent API allowing users to utilize such "external keys" across (at least) Windows and macOS. As a result, we'd like to build an abstraction over Windows Hello and macOS TouchID/Face ID.

We would start with a wrapper around the macOS keychain to support:

• generation, import, and destruction of RSA and EC key pairs
• authentication policies for keys (e.g. require a Touch ID fingerprint for every usage, ...)
• export of public keys for the stored private keys
• indicate storage location of keys (in a hardware secure element or the software keychain)
• usage of external keys for
  • Diffie-Hellman
  • Signing
  • Decryption

Further steps would include support for Windows Hello and a platform abstraction layer. For the latter it remains to be seen whether we need to converge on the least common denominator or not.

First obvious question: Would you be interested to provide such an API in Botan?

Furthermore, we'd like to start a discussion on an interface (or class hierarchy) for such an API. Especially, if and how the existing PKCS#11 and TPM wrappers would relate to this.

Here's a quick sketch of what we (@hrantzsch and @reneme) had in mind. It does feel somewhat over-object-oriented for our tastes, though.

[randombit]

Happy to support this - this is an integration that can be meaningfully used by many real world applications.

I would suggest an alternate design which would simplify your diagram. I won't have time to write it up fully until this evening but basic idea is that ExternalPrivateKey would be a completely new root class and not directly integrate with the existing hierarchy. You could then eg get the associated public key by calling std::unique_ptr<PublicKey> public_key() const.

Having PKCS11, TPM, and this integration all be distinct is really unfortunate and it's worth spending some time figuring out if we can get a design here that allows eventually converging the PKCS11/TPM support into it. Not that the existing TPM support is super relevant anymore as it is only 1.2 anyway.

[reneme]

ExternalPrivateKey would be a completely new root class and not directly integrate with the existing hierarchy.

Wouldn't that exclude the convenience of using those external private keys for existing algorithms, though? E.g. storing a TLS server's private key "externally" and just using the new implementation as a handle into Botan's TLS stack?

[randombit]

You're right that is a problem. I should explain first my motivations here re design.

a) The fact that Private_Key derives from Public_Key was a huge design mistake. It is true of course that an RSA private key is convertible to a RSA public key, but it's not an "is a" relationship. It makes much of the code more complicated, especially the inheritence hierarchy, and it becomes more complicated in cases where the private key is in one place (PKCS11, TPM) but public keys are not supported in the same place. You can see the same problem reflected in your proposed hierarchy.

b) The fact that low level types such as RSA_PrivateKey are exposed is another mistake. The API should have been expressed via only higher level base classes, and then these algorithm-specific APIs are internal. That is beneficial in terms of reducing API and ABI surface. The same problem was previously also applied to block ciphers etc and in 3.0 we're moving away from that were all access is via the base. However that's not quite possible yet for the PK code because there are certain operations which still cannot be viably accomplished using only the base classes.

I don't think this external key support needs to fix the above issues, but ideally it would not make them worse eg with a proliferation of new (user API visible) key types.

As an additional aside I think it would be better for any algorithm-specific distinction (if any is truly required) to be done on the basis of implementation. That is instead of RSA_macOS_ExtPrivateKey and ECDSA_macOS_ExtPrivateKey both deriving from ExternalPrivateKey you would have ExternalPrivateKey_macOS and then RSA_macOS_ExtPrivateKey and ECDSA_macOS_ExtPrivateKey deriving from that. Since as I read the APIs almost all of them are very agnostic to the underlying algorithm.

[randombit]

Reopening, mouse went astray

[randombit]

Sorry I know you are probably waiting on more design feedback from me here. One thing I am thinking about is that at least in some cases the external keys are interfaced via a handle (eg a PKCS11 session). And we need some way of identifying those objects, using whatever convention the underlying system uses. Would it make sense to have something like

class ExternalKeyMaterialSource {
   virtual ~ExternalKeyMaterialSource() = default;

   class Identifier {
      static Identifier from_uuid(const UUID& uuid);
      static Identifier from_string(const std::string& string);
      static Identifier from_blob(const std::vector<uint8_t>& blob);
   private: 
      std::vector<uint8_t> m_blob;
   }

   virtual std::unique_ptr<ExternalPrivateKey> load_private_key(Identifier identifier) = 0;
   // possibly later also load_public_key, load_secret_key, ...
};

then usage is something like

ExternalKeyMaterialSource_PKCS11 src(p11_session);
auto id = ExternalKeyMaterialSource::Identifier::from_uuid("...");
auto key = src.load_private_key(id);
if(key)
  use_key(*key);

I would imagine different systems have different ways of identifying keys (integer index, UUID, etc). Here I assume we can just treat everything as bytes, and let the specific source interpret those raw bytes however makes sense with the interface it is using.

[reneme]

In our mental model the <algo>_<platform>_ExtPrivateKey classes would have been internal implementation detail anyway. Only the new classes ExternalPrivateKey and <algo>_ExternalPrivateKey would be part of the user-facing API.

Just to be sure, we are on the same page:

For (e.g.) block ciphers, the currently preferred interface is the BlockCipher base class that is instantiated via its factory method: BlockCipher::create("AES-128"). This method would return a unique_ptr<BlockCipher> that uses virtual methods to dispatch invocations to the concrete implementations without cluttering API and ABI. Right?

I see that it would be beneficial to move asymmetric key types in a similar direction and hide most implementation detail behind a dynamically dispatched common base API (i.e. PrivateKey and PublicKey). Is it safe to assume that we won't need any algorithm-specific functionality for some schemes in the API, though?

For example, external private keys will likely require some sort of pre-init configuration, both when they are initially generated and when instantiating them later on (as an example, see Apple's Key Generation Attributes). It would be great to reflect that in the API as well. How would you feel about a sort-of builder pattern?

// generate a new private key on some hardware unit
auto keyConfig =
    KeyConfig::generate("RSA")
        .length(2048)                                  // keylength for RSA
        .authentication(KeyAuthentication::Biometric)  // always require user-authentication
                                                       // via e.g. Apple TouchID or Windows Hello
        .storage(KeyStorage::Keychain)                 // allow storage in a software keychain
        .usage(KeyUsage::ENCRYPT)                      // set allowed key usages
        .usage(KeyUsage::SIGN)
        .label("some fancy botan RSA key");            // user-facing label (e.g. in macOS keychain)

auto key = ExternalPrivateKey::create(keyConfig);  // generate a new RSA key in the OS keychain
auto pubkey = key->getPublicKey();                 // get a handle to the respective public key
auto keyid  = key->externalHandle()                // obtain the persistent key ID in the keychain

// - - - - - - - - - - - - 

// instantiate an existing key from the keychain
auto keyConfig2 = KeyConfig::instantiate(keyid);  // simply load the previously created key
auto key2 = ExternalPrivateKey::create(keyConfig2);

Obviously, it remains to be seen which of those configuration features are actually supported by the different target platforms.