Proposal: Building block API for cryptography

[ItalyPaleAle]

Note: Crypto as in cryptography, NOT cryptocurrency!!

In what area(s)?

/area runtime

Describe the proposal

This is a proposal for a new building block for Dapr to allow developers to leverage cryptography in a SAFE and consistent way. Goal is to expose an API that allows developers to ask Dapr to perform operations such as encrypting and decrypting messages, and calculating and verifying digital signatures.

Modern applications make extensive use of cryptography, which, when implemented correctly, can make solutions safer even in case data is compromised. Even more, in certain cases the use of crypto is required to comply with industry regulations (think banking) or even with legal requirements (GDPR). However, leveraging cryptography is hard: developers need to pick the right algorithms and options, and need to learn the proper way to manage and protect keys. Additionally, there are operational complexities when teams want to limit who has access to cryptographic key material.

Organizations have been increasingly started to leverage tools and services to perform crypto outside of applications. Examples include services such as Azure Key Vault, AWS KMS, Google Cloud KMS, etc. Customers may also use on-prem HSM products like Thales Luna. While those products/services perform the same or very similar operations, their APIs are very different.

This is an area where Dapr can help. Just like we're offering an abstraction on top of secret stores, we can offer an abstraction layer on top key vaults.

Using this building block, developers would be able to perform cryptographic operations without having to access raw key material. We would also offer a selection of algorithms that are configured correctly and forbid the usage of unsafe algorithms and operations. Algorithms available will depend on what the backend vaults support, but in general developers should always find AES (encrypt/decrypt only) and RSA; when supported, we can offer also ChaCha20-Poly1305 (encrypt/decrypt only) and ECC with ECDSA or EdDSA (sign/verify only).

Benefits would allow:

• Making it easier for developers to perform cryptographic operations in a safe way. Dapr provides safeguards against using unsafe algorithms, or using algorithms with unsafe options.
• Keeping keys outside of applications. Applications never see key material, but can request the vault to perform operations with the keys.
• Allowing greater separation of concerns. By using external vaults, only authorized teams can access private/shared key materials.
• Simplify key management and key rotation. Keys are managed in the vault and outside of the application, and they can be rotated without needing the developers to be involved (or even without restarting the apps).
• Enabling better audit logging to monitor when operations are performed with keys in the vault.

The new building block would feature 7 APIs:

• /encrypt: encrypts arbitrary data using a key stored in the vault. It supports symmetric and asymmetric ciphers, depending on the type of key in use (and the types of keys supported by the vault).
• /decrypt: decrypts arbitrary data, performing the opposite of what /encrypt does.
• /wrapkey: wraps keys using other keys stored in the vault. This is exactly like encrypting data, but it expects inputs to be formatted as keys (for example formatted as JSON Web Key) and it exposes additional algorithms not available when encrypting general data (like AES-KW)
• /unwrapkey: un-wraps (decrypts) keys, performing the opposite of what /wrap does
• /sign: signs an arbitrary message using an asymmetric key stored in the vault (we could also consider offering HMAC here, using symmetric keys, although not widely supported by the vault services)
• /verify: verifies a digital signature over an arbitrary message, using an asymmetric key stored in the vault (same: we may be able to offer HMAC too)
• /getkey: this can be used only with asymmetric keys stored in the vault, and returns the public part of the key

Different components would be developed to perform those operations on supported backends such as the products/services listed above. Dapr would "translate" these calls into whatever format the backends require. Dapr never sees the private/shared keys, which remain safely stored inside the vaults.

Additionally, we could offer a "local" crypto component where keys are stored as Kubernetes secrets and cryptographic operations are performed within the Dapr sidecar. Although this is not as secure as using an external key vault, it still offers some benefits such as using standardized APIs and separation of concerns/roles with regards to key management.

[yaron2]

I support adding a cryptography API as it will hide a lot of complexity and provide best practices in an area where they are very needed.

I particularly find useful the "local" version of the component.

[georgestevens99]

I support a "Building block for cryptography". And I would like to add a feature to your list of features that seeks to make end-to-end message encryption really, really easy to use, for both the sender and receiver. Namely, hide all the details of message encryption from the developer to have the kind of end-to-end message encryption necessary in highly secure situations (used even with a locked down cluster).

The Defense In Depth security best practice now widely recommended demands multiple levels of security. Just like putting your jewels in a safe within your locked house. Just a locked house is not good enough to protect really high value items. You need more layers of security -- a safe, how about a guard dog, an alarm on your front door, an alarm on your safe, etc. -- to have a good Defense In Depth. Note that this is key when communicating from a service to a Dapr side car. There is no mTLS on that network hop and to compensate for that there needs to be a good end-to-end message encryption that can easily span this potential message security gap. Plus it needs to be really easy to use so that a developer does not have to understand all about public and private keys, etc., etc. to effectively do highly secure, failsafe, end-to-end message encryption on both the sending and receiving ends.

When using encrypted messages flowing between 2 services (that may be located who knows where, and not just locally, but it applies to a local cluster/host as well) end to end message encryption is highly desireable, especially for portions of the message that are seriously private. In order for the sender and receiver to encrypt/decrypt the message (or portions of it, like PII) they need to share encryption keys, aka the public key -- this the the hardest part of end-to-end encryption.

Please implement an end-to-end message encryption feature that uses a "Secure Conversation" concept that allows the sender and receiver to securely communicate to exchange info containing the encryption key for each message. There is a spec defined for this called WS-SecureConversation. Described as "Following a pattern similar to TLS, WS-SecureConversation establishes a kind of session key." See https://en.wikipedia.org/wiki/WS-SecureConversation and, for a complete spec http://docs.oasis-open.org/ws-sx/ws-secureconversation/v1.4/ws-secureconversation.html.

Plus, please make it really easy to use through a simple API which I will leave to others to specify in detail. It would be nice to have a single really high level operation like EndToEndEncrypt(someMessage) that takes care of all the grubby secure conversation/"session key" details, plus the encryption, behind the scenes.

The service would call EndToEndEncrypt() to encrypt the message before it is sent to message through Dapr via Service Invocation AND/OR Pubsub. Then Dapr would take care of all the rest of the details. Then when an encrypted message is received by the receiver Dapr does the decryption behind the scenes as well.

I believe that such a "session key" may be able to be obtained from Sentry, or based on things that Sentry already does. Thus, this feature, especially the "session key" part of it, may not be all that difficult to implement, leveraging existing code. (I hope).

Thanks for the highly useful idea!

[ItalyPaleAle]

Hey @georgestevens99 thanks for sharing your thoughts!

I would like to understand more about your requirements for E2E encryption in this case.

As you know, communication between Dapr-ized applications already uses TLS. Between Dapr sidecars, communications use gRPC with mTLS, which offers network-level encryption and mutual authentication. Communication between applications and the Dapr sidecar isn't encrypted, but that should happen locally.

Because of that, I don't know if the requirement for providing an API for E2EE communication would fit in the scope of this proposal, which is more focused on lower-level cryptographic primitives than higher-level, opinionated protocols.

If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). I don't know if this necessarily needs to be implemented in Dapr itself, however: I feel that for true E2EE you would want to implement this in your applications: each service would expose an endpoint that is invoked to perform the ECDH agreement, and that allows obtaining the shared secret. This would happen with a regular service-to-service invocation between your applications, which would transparently pass through Dapr (ECDH is designed to be used over insecure channels so it's perfectly safe even if no encryption is present). If this were implemented within Dapr, additionally, then the sidecars would be able to sniff the shared encryption key, thus violating the E2EE requirement.

Perhaps we could provide an example of how to do this design (ECDH to derive a shared encryption key) even with current versions of Dapr :)

[georgestevens99]

E2EE for a message (or parts of it) is suggested by the dapr documents for highly confidential info. But dapr currently supplies no way to do such encryption.

While E2EE can certainly be done at a low level as is your idea, the fact of the matter is that there are vast numbers of developers that do not have the experience, knowledge, nor skills to use the "lower level crypto privatives" you are proposing. Couple that with the fact that security is growing more and more challenging each year. Thus in addition to having lower level primitives for those who have the skills to use them, we (our industry) also need a really, really simple to use and highly effective and highly secure E2EE for messages sent from a Service to a Dapr sidecar and beyond. This will provide a solid, highly productive solution for the encryption for this network hop (and beyond) that is suggested in the Dapr docs.

In my experience, most developers haven't a clue as to what you said means, i.e. "If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). ". And learning how to effectively use this low level stuff is non trivial and error prone and takes time.

So we need E2EE for developers that have little or no background in security. So think of it as a higher level addition to your excellent proposal. This high level E2EE should be as easy to use as is Dapr Service Invocation or Pubsub or State Management is when done via one of the SDKs. Then, anyone who can write code with Dapr can also do near-world class E2EE of their messages and PII without climbing a steep security learning curve. That is the overall goal of my feature proposal.

[ItalyPaleAle]

A fully-transparent E2EE would only be possible between Dapr sidecars, but Dapr sidecars already talk to each other using mTLS (which is enabled by default), which should provide equal levels of security and authentication.

Perhaps a middle-ground would be to implement an API that performs the ECDH between two services (technically, between two sidecars) and gives the application a shared secret they can then use as they see fit, including using the /encrypt endpoint of this proposal. However it should be clear that in this case the sidecars do have visibility over the encryption keys, even if for just an instant, so it's not true E2EE between the applications. I am not sure if that would void some of the security requirements you're talking about.

Or, we could just create some samples that demonstrate how to do what I described above: creating an endpoint in a service so that they can perform an ECDH agreement, then use that shared secret with the /encrypt endpoint.

Also worth pointing out that in all the cases above, both with things that are integrated with Dapr or implemented in the applications, there's always the problem of authenticating the other party. Dapr doesn't authenticate application IDs, so anyone could create an application with any ID: just because you're performing an ECDH with an application called foo, you don't have any guarantee that it's the foo you want to talk to.

[georgestevens99]

@ItalyPaleAle -- Thanks so much for taking the time to write up the various alternatives that aim at fulfilling my requrements. I really appreciate it. It forced me to think through my requirements and also the limitations inherent in Dapr as it now stands.

I was hoping there was a way to use Dapr's future capabilities to implement industrial strength E2EE for messages. However, given the caveats you pointed out in your text (many thanks for those!) I am abandoning my attempts to use Dapr for this. Instead I am "biting the bullet" and planning on developing a small Security Utility service to at least deal with the encryption keys for the sender and receiver, and likely more. That way I can use identity based security to authenticate the sender and receiver to the Security Utility to setup encryption keys and avoid the potential identity ambiguity you described in your last paragraph, plus also avoid a tiny "instant" you described where "the sidecars do have visibility over the encryption keys", thus resulting in better overall security.

Thus, I am withdrawing my request for extending your "Building block for cryptography" to support E2EE as well.

Based on our conversation thus far, it seems to me that Dapr could offer such a standalone, un-daprized (no sidecar) Security Utility for E2EE. This would be highly useful to those of us who need to deal with very high levels of security. Any thoughts on the possibility of a such a Dapr Security Utility?

Thanks again for your contribution to this conversation!

[ItalyPaleAle]

Thanks for the detailed writeup!

I think what you wrote could be interesting. I don't know if this would be part of the "official" Dapr or some "community-contributed" extension.

The biggest problem about making this official is that there may not be a one-size-fits-all. So we'd have to be opinionated and with that we'd have to make tradeoffs.

Perhaps we should take this to another issue (or on discord). It would be useful to know your requirements too. Establishing a secure communication channel is "easy" (I described above how it could just be as "simple" as using ECIES, by means of performing an ECDH over Dapr's service invocation APIs). But if the requirements include authenticating services, then you start getting into needing a PKI or something like that (how 'bout a blockchain? 😂 just kidding). Things can spiral into getting very complicated, very fast.

[georgestevens99]

@ItalyPaleAle --Good idea. I am OK on being opinionated and making tradeoffs. I am mainly interested in a E2EE security utility that works with .NET, especially ASP.NET Core gRPC services using Protobuf-net code first, and using either certificate or AAD identity based authentication with each service having its own identity. But within that package there is core functionality that could probably be used by various languages, etc. And I am interested in preventing complexity spirals as well.

What Dapr Repo would be appropriate for a "Standalone Dapr E2EE Security Utility"? Lets start a new issue for that in the appropriate repo and take it from there!

[olitomlinson]

+1 for this proposal.

Particularly interested in this building block, as we want to offer a Bring Your Own Keys feature to our customers.

It would be super neat if a tenant in our multi-tenant system could share with us their key (one-time, super locked down SDLC for this part of the app which receives keys) and then we never get to see that key ever again because it is tucked away in a cloud vendor HSM/daprd, and our developers never get access to that key as they use an API something like this :

I would see a dapr API looking something like this

POST http://localhost:3500/v1.0/crypto/{tenant_id}/encrypt

REQ

{
  "unencrypted_payload" : "I am super secret tenant data. Its encrypted state will be used downstream in dapr State Management & dapr Pub Sub components"
}

RESPONSE

{
  "encrypted_payload" : "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y"
}

downstream....

POST http://localhost:3500/v1.0/state/someStateComponent

[
  {
    "key": "weapon",
    "value": "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y",
    "etag": "1234"
  }
]

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[ItalyPaleAle]

👋🤖

[olitomlinson]

@ItalyPaleAle Has there been any further discourse behind the scenes around moving this proposal forward? Cheers!

[artursouza]

Not yet. We would welcome a POC + proposal from the community.

[joelyen]

We have internal application encryption requirements for our project, and we would like to standardize the encryption operation across all the services. Since this new building block won't be available for a while, does it make sense for us to internally build the "local" version following the Output Binding Component spec?

[ItalyPaleAle]

@joelyen the biggest blocker to this is that we need to implement support for streaming first. These APIs need to be streaming-first so they can handle larger amounts of data without issue. You can try building this with the output bindings spec, but you may have issues if trying to process larger blobs of data.

Which operations are you specifically interested in?

[artursouza]

We discussed this issue in the community call today. Here goes the general feedback on it (including some of my recommendations too):

1. Encrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/encrypt
Payload goes directly into the body.

Response: RAW encrypted payload in the HTTP response body, passing response metadata (encryptedKey) via Headers Alternative: Return the Dapr's encryption envelope as JSON

{
  "ciphertext": "AJGIGBKJGH",
  "encryptedKey": "BJKHGIUXCBKLDLK"
}

2. Decrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/decrypt
Response body from `encrypt` goes directly into this HTTP request body and propagate Headers too since `encryptedKey` is required to decrypt.

Response: RAW plaintext in the HTTP response body.

3. Built-in encryption component

• Cipher information is part of the component
• Fetches encryption key from secret store

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: mycipher
spec:
  type: crypto.builtin
  version: v1
  metadata:
  - name: cipher
    value: AES256
  - name: secretStore
    value: mysecretstore
  - name: encryptionKeyName
    value: mysymmkey

4. Encryption components

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: vault
spec:
  type: crypto.hashicorp.vault
  version: v1
  metadata:
  - name: vaultAddr
    value: "https://127.0.0.1:8200"
  - name: encryptionPath
    value: /transit/encrypt/mykey

5. Dapr encryption envelope Dapr runtime will have an opinion on how to encrypt messages and minimize data sent/received from encryption services. It means that the Dapr runtime will generate on-time encryption keys and encrypt those with services instead of encrypting the whole payload. The payload is then encrypted with the on-time encryption key instead. The Dapr's encryption envelop will contain the ciphertext and the encrypted key (at least). The envelope can be split into body and headers in the encrypt response or serialized as JSON.

[tmacam]

I think we need a way to also make envelope encryption exposed through Dapr. Some cloud providers (AWS, IBM) expose envelope encryption as a first class concept. Additionally, it is a common pattern for enabling secure high-throughput client side encryption. It might be a worth concept to have as a building block for distributed apps exposed through Dapr.

Context

So what is envelop encryption?[1], [2], [3]

Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.

Why even doing this? From the KMS Tutorial (GCP docs for envelope encryption is also pretty good):

AWS KMS can encrypt or decrypt data up to 4 KB in length. This can make it difficult to work with larger pieces of data. Additionally, when you encrypt data, it must be transferred over the network, processed by KMS, and returned; often, the associated network load and latency and complexity makes this undesirable.

A popular alternative to encrypting data directly with KMS is the use of envelope encryption . Rather than sending the data to KMS to be encrypted, we use KMS to generate a data key, which is then encrypted with a KMS-managed master key, or CMK. We then use this key - not the CMK itself - to encrypt and decrypt data, locally, where we are processing the data.

So the idea of envelope encryption is that

1. You generate a random key per encryption request to encrypt your data (your data encryption key or DEK). Data of any size can be encrypted locally, fast and unbounded by your cloud provided limits/quotas.
2. You encrypt your data with this DEK.
3. You encrypt this DEK with a key stored in your key management service, in a process called key wrapping. That will be your Key Encryption Key (or CMK, depending on the terminology 😉 ). Multiple DEK can be encrypted with the same KEK.
4. The DEK along with the data it wrapped (encrypted) are stored/sent together.

Why envelope encryption?

1. Reduced costs when working for cloud services - although some providers+SDKs will charge you for envelop encryption, the costs of using envelop encryption are lower than requesting keys and encryption of all data directly to the cloud provider.
2. Resiliency and increased TPS - cloud providers have limits on the rate of encryption operations one can requests and amount of data requests. Envelop encryption reduces those.
3. Decreased latency - encrypted data doesn't have to leave premises and are encrypted locally, saving transfer and latency fees
4. Consistency - cloud providers already offer this. By not offering an equivalent to this on Dapr we alienate those customers and decrease Dapr usefulness on this regard.

Support by cloud provider and encruption

• AWS exposes envelope encryption as a first class concept in their SDK.

• Azure Key Vault doesn't expose this directly in its APIs. It is handled transparently with its managed services and performed by its client libraries for storage

• GCP doesn't explicitly offer envelop encryption as a first class concept in their SDK but its Tink Cryptographic library exposes it. That said, envelope encryption is the under-the-hood solution for encryption data with asymmetric cryptography (but that is expected).

• IBM cloud exposes envelop encryption as an API

• Hashicorp Vault transit backend also offers primitives for generating DEK through its generate-data-key operation. src

Why we need Dapr support? Can't users implement this themselves?

One thing to keep in mind regarding envelop encryption is that when one performs the key wrapping step, metadata about the KEK used (version, name etc) has to be persisted together with the (now encrypted) DEK. This is required so so KEK rotation can be performed without impacting already issued DEK and data they encrypt. KEK metadata is not something we are exposing to clients in this proposal. So in order to offer client-side encryption using envelop encryption we would need to do it on the component side.

Again, some SDKs expose this as a first class concept. If we don't offer the right abstraction for this customers will be barred from this.

EDITS

• Section "Why we need Dapr support
• Note about vault transit backend having support for envelope encryption

[artursouza]

The proposal from @tmacam goes into details on how Dapr should have an opinion on the encryption envelope I mentioned earlier. Thanks, @tmacam!

[tmacam]

After re-reading this proposal I noticed support for envelope encryption is present in the form of /wrapkey and /unwrapkey APIs. I do wonder if it makes sense to offer an abstraction of client-side encryption using envelope encryption under the hood.

[ItalyPaleAle]

Hey folks, I wanted to write down my thoughts next week once I’m back from vacation, but I had given this a bit more thought–and it also addresses @tmacam’s feedback.

I can flush this out in a full API design next week, but at a high level…

——

First, my original proposal was for exposing low-level primitives only. I think we still need to do that, and in addition to methods to encrypt/decrypt data with symmetric ciphers (what @artursouza already proposed), we also need:

• Methods to encrypt and decrypt data with asymmetric ciphers
• Methods to wrap/unwrap keys - this is essentially the “envelope encryption” @tmacam is talking about. These methods are encryption/decryption routines, but they allow for different encryption schemes that do not make sense when encrypting arbitrary messages.
• Methods to sign data and verify signatures

I flushed these things out a bit more in the first post of this issue, with a bit more details.

(We could also consider offering methods to calculate hashes/digests, although those may be less important as they’re fairly common)

Second, I do think that we can provide additional value besides just offering these low-level primitives to implement on top of them higher-level APIs that implement common encryption schemes that make it easier for people who aren’t versed in the subject to encrypt and decrypt data safely.

This is aligned with what @tmacam is talking about too, as that’s getting into the realm of hybrid encryption, for example.

Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)

——

To sum things up, I think we should offer 2 sets of APIs:

• /subtlecrypto offers the low-level primitives to encrypt/decrypt, sign/verify, wrap/unwrap keys. These are low-level functions that offer direct access to the ciphers and require a certain level of understanding of what one is doing. I like the “subtle” term, which I copied from the JS WebCrypto APIs, because it implies that it’s something to be treated with care. The MDN docs linked there have a nice warning along the lines of “this stuff is dangerous and you need to know what you’re doing”
• /crypto would offer higher level, opinionated APIs to encrypt and decrypt data (no signatures or key wrapping) using a limited set of encryption schemes, including hybrid ones (e.g. RSA+AES).

From an implementation point of view, the higher-level APIs invoke the subtle APIs internally, and are only implemented in the runtime: no work needed on components-contrib, where only the low-level APIs are offered.

There are also a few more things that I would like to point out:

1. These APIs can deal with large amounts of data so support for streaming (which requires #4979) in the runtime will be essentially mandatory - lest users will be sending GBs of data to Dapr
2. While streaming is not a problem for most operations, we need to be careful in certain cases, such as when decrypting data that was encrypted with an authenticated cipher (such as AES-GCM, ChaCha20-Poly1305, or AES-CBC+SHA256), because we need to decrypt the entire payload before we can send it to the client, in case the authentication tag check fails (the authentication tag can only be computed only when the entire message has been processed, and we should not flush un-authenticated data to the client). For higher-level APIs we could get around this by chunking data and adding authentication tags periodically (see for example the DARE scheme)

We also need to agree on what algorithms we support and what modes of operation. I suggest (not all of them may be available if the key vaults don’t support them):

• Symmetric ciphers: AES-GCM, AES-CBC, AES-CBC+SHA256, ChaCha20-Poly1305. For AES, we support keys 128, 192, and 256-bits long. For key wrapping, there’s also AES-KW
• Asymmetric ciphers for data encipherment: RSA-OAEP (2048, 3072, 4096) - we should not support the old RSA defined in PKCS#1 v1.5
• Signatures: RSA-PSS (PKCS#1 v2.1) and RSA-PKCS#1 v1.5, ECDSA (NIST curves), EdDSA (Curve25519 at least)

For the higher-level APIs, we should discuss what schemes we want to support, especially with regards to hybrid encryption.

[artursouza]

Hey folks, I wanted to write down my thoughts next week once I’m back from vacation, but I had given this a bit more thought–and it also addresses @tmacam’s feedback.

I can flush this out in a full API design next week, but at a high level…

——

First, my original proposal was for exposing low-level primitives only. I think we still need to do that, and in addition to methods to encrypt/decrypt data with symmetric ciphers (what @artursouza already proposed), we also need:

• Methods to encrypt and decrypt data with asymmetric ciphers
• Methods to wrap/unwrap keys - this is essentially the “envelope encryption” @tmacam is talking about. These methods are encryption/decryption routines, but they allow for different encryption schemes that do not make sense when encrypting arbitrary messages.
• Methods to sign data and verify signatures

I flushed these things out a bit more in the first post of this issue, with a bit more details.

(We could also consider offering methods to calculate hashes/digests, although those may be less important as they’re fairly common)

Second, I do think that we can provide additional value besides just offering these low-level primitives to implement on top of them higher-level APIs that implement common encryption schemes that make it easier for people who aren’t versed in the subject to encrypt and decrypt data safely.

This is aligned with what @tmacam is talking about too, as that’s getting into the realm of hybrid encryption, for example.

Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)

——

To sum things up, I think we should offer 2 sets of APIs:

• /subtlecrypto offers the low-level primitives to encrypt/decrypt, sign/verify, wrap/unwrap keys. These are low-level functions that offer direct access to the ciphers and require a certain level of understanding of what one is doing. I like the “subtle” term, which I copied from the JS WebCrypto APIs, because it implies that it’s something to be treated with care. The MDN docs linked there have a nice warning along the lines of “this stuff is dangerous and you need to know what you’re doing”
• /crypto would offer higher level, opinionated APIs to encrypt and decrypt data (no signatures or key wrapping) using a limited set of encryption schemes, including hybrid ones (e.g. RSA+AES).

From an implementation point of view, the higher-level APIs invoke the subtle APIs internally, and are only implemented in the runtime: no work needed on components-contrib, where only the low-level APIs are offered.

There are also a few more things that I would like to point out:

1. These APIs can deal with large amounts of data so support for streaming (which requires Migrate HTTP server from fasthttp to net/http #4979) in the runtime will be essentially mandatory - lest users will be sending GBs of data to Dapr
2. While streaming is not a problem for most operations, we need to be careful in certain cases, such as when decrypting data that was encrypted with an authenticated cipher (such as AES-GCM, ChaCha20-Poly1305, or AES-CBC+SHA256), because we need to decrypt the entire payload before we can send it to the client, in case the authentication tag check fails (the authentication tag can only be computed only when the entire message has been processed, and we should not flush un-authenticated data to the client). For higher-level APIs we could get around this by chunking data and adding authentication tags periodically (see for example the DARE scheme)

We also need to agree on what algorithms we support and what modes of operation. I suggest (not all of them may be available if the key vaults don’t support them):

• Symmetric ciphers: AES-GCM, AES-CBC, AES-CBC+SHA256, ChaCha20-Poly1305. For AES, we support keys 128, 192, and 256-bits long. For key wrapping, there’s also AES-KW
• Asymmetric ciphers for data encipherment: RSA-OAEP (2048, 3072, 4096) - we should not support the old RSA defined in PKCS#1 v1.5
• Signatures: RSA-PSS (PKCS#1 v2.1) and RSA-PKCS#1 v1.5, ECDSA (NIST curves), EdDSA (Curve25519 at least)

For the higher-level APIs, we should discuss what schemes we want to support, especially with regards to hybrid encryption.

I am not sure about offering the low level crypto APIs because we will lose the portability cross components. We should have a common denominator API that is compatible across cloud services first. The more we add to the request, the less portable it is. This was one of the outcomes of the discussion in the community call. /cc @yaron2

IMO, we should have /encrypt /decrypt /sign /verify but not /wrap /unwrap and not have any configuration via the request and move as much as possible to the component config instead.

[yaron2]

I am not sure about offering the low level crypto APIs because we will lose the portability cross components. We should have a common denominator API that is compatible across cloud services first. The more we add to the request, the less portable it is. This was one of the outcomes of the discussion in the community call. /cc @yaron2

IMO, we should have /encrypt /decrypt /sign /verify but not /wrap /unwrap and not have any configuration via the request and move as much as possible to the component config instead.

Agree, we should focus on portability and the scenarios that can be covered by the built-in component type. I value the portability aspect a lot and consider it the as one of the first points of optimization for any given design.

We can always add "lower level" APIs/endpoints later on.

[ItalyPaleAle]

I wasn't in the community call, but I am not sure I understand the concerns about portability.

All key vaults expose only crypto primitives as the lowest common denominator. Although some algorithms aren't universal, we can count on a core set to be present, including AES and RSA (AFAIK ChaCha20 isn't available anywhere, and support for ECDSA/EdDSA is limited).

When we implement this in Dapr, the most convenient thing (easier to maintain and less error-prone) would be to implement only the crypto primitives in the components. For example, implement methods to perform AES encryption/decryption with a given mode of operation and no further abstractions.
Additional abstractions should be implemented in the runtime instead to ensure that the behavior is consistent across all components (and portable, reliable, safe).

Higher level APIs are implemented in the runtime and interact with the components by leveraging the primitives.
For example, if we decide that we will offer a higher-level API to encrypt data that is opinionated, we can decide that it will work by encrypting data with an ephemeral key (data encryption key or DEK), which is then wrapped with the user key (key encryption key or KEK) stored in the vault. The flow would be:

• runtime receives the request to encrypt data
• runtime generates a random DEK
• runtime invokes the component to wrap the DEK using the key stored in the vault - this is the only operation that needs to leverage the vault
• runtime encrypts the data using the DEK and returns the wrapped DEK and the ciphertext to the client - it doesn't make sense to do this in the vault because of performance (need to send the entire data to the vault over the network) and cost

I don't think there's another way to implement such a routine in a portable and safe way.

Given the above, it probably makes sense to also expose the primitives directly, for example as "subtlecrypto", as they would already be implemented by the components. We can use the component metadata API to expose what algorithms are supported should people need "non-core" ones.

Access to the lower level APIs will also be a requirement for people who need compatibility with existing or external systems, as people may need to implement their own schemes. For example, imagine encrypting data in the server (with Dapr) and then having to decrypt it in a client (using JS).

This is for encryption/decryption. Computing and verifying signatures are by their own nature "subtle" because there are no commonly-used (or needed) higher level schemes.

[artursouza]

I wasn't in the community call, but I am not sure I understand the concerns about portability.

All key vaults expose only crypto primitives as the lowest common denominator. Although some algorithms aren't universal, we can count on a core set to be present, including AES and RSA (AFAIK ChaCha20 isn't available anywhere, and support for ECDSA/EdDSA is limited).

When we implement this in Dapr, the most convenient thing (easier to maintain and less error-prone) would be to implement only the crypto primitives in the components. For example, implement methods to perform AES encryption/decryption with a given mode of operation and no further abstractions. Additional abstractions should be implemented in the runtime instead to ensure that the behavior is consistent across all components (and portable, reliable, safe).

Higher level APIs are implemented in the runtime and interact with the components by leveraging the primitives. For example, if we decide that we will offer a higher-level API to encrypt data that is opinionated, we can decide that it will work by encrypting data with an ephemeral key (data encryption key or DEK), which is then wrapped with the user key (key encryption key or KEK) stored in the vault. The flow would be:

• runtime receives the request to encrypt data
• runtime generates a random DEK
• runtime invokes the component to wrap the DEK using the key stored in the vault - this is the only operation that needs to leverage the vault
• runtime encrypts the data using the DEK and returns the wrapped DEK and the ciphertext to the client - it doesn't make sense to do this in the vault because of performance (need to send the entire data to the vault over the network) and cost

I don't think there's another way to implement such a routine in a portable and safe way.

Given the above, it probably makes sense to also expose the primitives directly, for example as "subtlecrypto", as they would already be implemented by the components. We can use the component metadata API to expose what algorithms are supported should people need "non-core" ones.

Access to the lower level APIs will also be a requirement for people who need compatibility with existing or external systems, as people may need to implement their own schemes. For example, imagine encrypting data in the server (with Dapr) and then having to decrypt it in a client (using JS).

This is for encryption/decryption. Computing and verifying signatures are by their own nature "subtle" because there are no commonly-used (or needed) higher level schemes.

I have used a "crypto API" before and the algorithm used was outside the code. It was defined when creating the encryption key in the encryption service. This way, the app just needs to encrypt passing the payload and keyname. I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

[ItalyPaleAle]

I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

The value Dapr provides is that it abstracts the various cloud key vaults. All libraries that offer the primitives expect the key material to be passed directly.

[vikas2k14]

I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

The value Dapr provides is that it abstracts the various cloud key vaults. All libraries that offer the primitives expect the key material to be passed directly.

I support this...we are also using dapr in all our applications and this is one of our requirements...if dapr could orchestrate that that would be really helpful.

We have one service for all the data security-related requirement ( like encryption/decryption/signing XML) so it would be really helpful if we could orchestrate the backend kms system using dapr.

[ItalyPaleAle]

API design proposal

High-level vs subtle

The building block features 2 kinds of operations:

• Low-level or "subtle" (term frequently used to indicate low-level crypto operations; one example is how in browsers, low-level operations are in the crypto.subtle object). These offer developers full control over the schemes that are used, allowing them to specify parameters, keys, modes of operations, etc.
  As the "subtle" name implies, using these operations requires a certain level of understanding about what they do and how to use them safely; they are not meant to be consumed by the "general public".
  Dapr offers these low-level operations for two reasons:
  1. Just like it is Dapr's value proposition to, for example, offer a consistent API abstracting various state stores, we will offer a consistent API to interface with key vaults. It allows using key vaults from multiple cloud providers, as well as using keys stored as Kubernetes secrets, all while interfacing with the same API.
  2. Developers working on applications that require interacting with existing and/or external solutions need these lower-level APIs to be able to maintain compatibility with them.
• High-level operations. At launch, these will cover data encryption and decryption only.
  When using these higher-leel methods, Dapr offers a great level of abstraction. Developers just need to provide a key (symmetric or asymmetric) and can then use Dapr to encrypt/decrypt data without having to worry about anything else. Dapr will choose the best ciphers and modes of operations, offering access to an encryption scheme that is secure and flexible.

Dapr encryption scheme: dapr.io/enc/v1

In the first version of the building block, we define 2 higher level operations to encrypt and decrypt data, in addition to low-level operations.

Sources: The encryption scheme that Dapr uses is heavily inspired by the Tink wire format (from the Tink library maintained by Google), as well as by Filippo Valsorda's age, and Minio's DARE.

The Dapr encryption scheme is optimized for processing data as a stream. Data is chunked into multiple parts which are encrypted independently. This allows us to return data to callers as a stream, even when decrypting messages, being confident that we are not flushing unverified data to the client.

Key

Each message is encrypted with a 256-bit symmetric File Key (FK) that is randomly generated by Dapr for each new message. The key must be generated as 32 byte of output from a CSPRNG (such as Go's crypto/rand.Reader) and must not be reused for other files.

The FK is wrapped using a key stored in a key vault (Key Encryption Key (KEK)) by Dapr. The result of the wrapping operation is the Wrapped File Key (WFK). The algorithm used depends on the type of the KEK as well as the algorithms supported by the component: in order of preference:

• Symmetric keys:
  • AES-KW (RFC 3394): AES-KW
  • AES-GCM: AES-GCM
    The result of using AES-GCM is in the format iv || wfk || tag
• RSA keys:
  • RSA OAEP with SHA-256: RSA-OAEP-256

In the future, we should explore how to add support for elliptic curve cryptography, for example Curve25519, which requires performing an ECDH key agreement.

Ciphertext format

The ciphertext is formatted as:

header || binary payload

Header

The header is human-readable and contains 4 items, each terminated by a line feed (0x0A) character:

1. Name and version of the encryption scheme used. Currently, this is always dapr.io/enc/v1
2. The algorithm used to wrap the file key, as well as any other parameters as needed (currently unused, but will be needed for ECDH)
3. The wrapped file key, base64-encoded
4. The MAC for the header, base64-encoded

Base64 encoding follows RFC 4648 §4 ("standard" format, with padding included but optional when decoding)

dapr.io/enc/v1
AES-KW
hGYjwDpWEXEymSTFZ95zgX8krElb3Gqyls67R8zJA3k=
pBDKLrhAWL7IAvDKBV/v7lmbTG6AEZbf3srUN0Pnn30=

The final line is the MAC for the header, which is computed with HMAC-SHA-256 over the previous 3 lines (including the final newline) with a key that is derived from the (plain-text) File Key with HKDF-SHA-256:

mac-key = HKDF-SHA-256(ikm = file key, salt = empty, info = "header")
MAC = HMAC-SHA-256(key = mac-key, message = first 3 lines of the header)

Note that there's one newline character (0x0A) at the end of the MAC, which concludes the header

Binary payload

The binary payload begins immediately after the header (after the 4th newline character) and it includes the binary header and each segment of data encrypted:

binary header || segment_0 || segment_1 || ... || segment_k 

Binary header

The binary header is 8-bytes long:

Cipher suite	Nonce prefix
1 byte	7 bytes
0x00: AES-GCM 0x01: ChaCha20-Poly1305 (RFC 8439)	Random sequence of 7 bytes generated by a CSPRNG

Dapr will choose AES-GCM if running on a system that supports AES-NI CPU instructions, or ChaCha20-Poly1305 otherwise. We may change that behavior in the future.

Segments

The plaintext is chunked into segments 64KB (65,536 bytes) each; the last segment may be shorter. Segments must never be empty, unless the entire file is empty.

Each segment of plaintext is encrypted independently and stored together with its tag:

encrypted chunk || tag

Segments are encrypted with a Payload Key (PK) that is derived from the (plain-text) File Key and the nonce prefix:

payload-key = HKDF-SHA-256(ikm = file key, salt = nonce prefix, info = "payload")

Each segment is encrypted using a different 12-byte nonce / initialization vector:

nonce_prefix || i || last_segment

Where:

• nonce_prefix (7 bytes) is the nonce prefix from the binary header
• i (4 bytes) is the sequence number, as a 32-bit unsigned integer counter, encoded as big-endian. The first segment has sequence number 0, and it increases.
• last_segment (1 byte) is 0x01 if this is the last segment, or 0x00 otherwise

Components

Components in dapr/components-contrib implement low-level primitives only, while all higher-level operations are performed by the runtime, so they are executed in a consistent way across all backends/services. This is because the job of the components is limited to actually interacting with the key vaults, and everything else is best handled by the runtime.

Components are to be placed in the crypto folder and must implement the SubtleCrypto interface:

import "github.com/lestrrat-go/jwx/v2/jwk"

// SubtleCrypto offers an interface to perform low-level ("subtle") cryptographic operations with keys stored in the vault
type SubtleCrypto interface {
    // GetKey returns the public part of a key stored in the vault
    // This method returns an error if the key is symmetric
    GetKey(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Name (or name/version) of the key to use in the key vault
        key string,
    ) (
        // Object containing the public key
        pubKey jwk.Key,
        // Error
        err error,
    )

    // Encrypt a small message and returns the ciphertext
    Encrypt(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Input plaintext
        plaintext []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Encrypted ciphertext
        ciphertext []byte,
        // Authentication tag
        // This is nil when not using an authenticated cipher
        tag []byte,
        // Error
        err error,
    )

    // Decrypt a small message and returns the plaintext
    Decrypt(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Input ciphertext
        ciphertext []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Authentication tag
        // Ignored when not using an authenticated cipher
        tag []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Decrypted plaintext
        plaintext []byte,
        // Error
        err error,
    )

    // WrapKey wraps a key
    WrapKey(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Key to wrap as jwk.Key object
        plaintextKey jwk.Key,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Wrapped key
        wrappedKey []byte,
        // Authentication tag
        // This is nil when not using an authenticated cipher
        tag []byte,
        // Error
        err error,
    )

    // UnwrapKey unwraps a key
    UnwrapKey(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Wrapped key
        wrappedKey []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Authentication tag
        // Ignored when not using an authenticated cipher
        tag []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Plaintext key
        plaintextKey jwk.Key,
        // Error
        err error,
    )

    // Sign a digest
    Sign(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Digest to sign
        digest []byte,
        // Signing algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        // The key must be asymmetric
        key string,
    ) (
        // Signature that was computed
        signature []byte,
        // Error
        err error,
    )

    // Verify a signature
    Verify(
        // Context that can be used to cancel the running operation
        ctx context.Context, 
        // Digest of the message
        digest []byte,
        // Signature to verify
        signature []byte,
        // Signing algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        // The key must be asymmetric
        key string,
    ) (
        // True if the signature is valid
        valid bool,
        // Error
        err error,
    )
}

A few notes about all methods above:

1. Keys are passed as jwk.Key objects, from the (excellent) lestrrat-go/jwx library
2. The algorithm should be represented as constant as defined by RFC 7518 ("JSON Web Algorithms (JWA)"). For the most part, Dapr components should not try to parse the value submitted by the user (unless the component is the "local" one that performs crypto operations directly), and pass whatever value directly to the key vault.
3. The key parameter can contain a version if keys can be versioned in the vault. The format should be name/version. If no version is specified, it's assumed to be the latest.

Notes on WrapKey and UnwrapKey:

1. If the key need to be encoded (common with asymmetric keys), it needs to be encoded before being passed to the component. For example, in the runtime, RSA keys may be represented in a rsa.PrivateKey object, and need to be encoded in PKCS#1 format. Symmetric keys can be passed as-is, as they are normally stored in a byte slice already.
2. WrapKey and UnwrapKey can be implemented on top of Encrypt and Decrypt if the underlying key vault does not have a special operation for key wrapping/unwrapping.

Notes on Encrypt and Verify:

1. When using an asymmetric key, these operations can be performed using the public key without hitting the key vault. However, for consistency and to ensure that we always use the last version of the key, components should always perform them in the vault. Exception could be if the key has a specific version, in which case components may opt to download the public key, cache it, and perform the operation locally.

gRPC APIs

In the Dapr gRPC APIs, we are extending the runtime.v1.Dapr service to add new methods:

// (Existing Dapr service)
service Dapr {
  // SubtleGetKey returns the public part of an asymmetric key stored in the vault.
  rpc SubtleGetKey(SubtleGetKeyRequest) returns (SubtleGetKeyResponse);

  // SubtleEncrypt encrypts a small message using a key stored in the vault.
  rpc SubtleEncrypt(SubtleEncryptRequest) returns (SubtleEncryptResponse);

  // SubtleDecrypt decrypts a small message using a key stored in the vault.
  rpc SubtleDecrypt(SubtleDecryptRequest) returns (SubtleDecryptResponse);

  // SubtleWrapKey wraps a key using a key stored in the vault.
  rpc SubtleWrapKey(SubtleWrapKeyRequest) returns (SubtleWrapKeyResponse);

  // SubtleUnwrapKey unwraps a key using a key stored in the vault.
  rpc SubtleUnwrapKey(SubtleUnwrapKeyRequest) returns (SubtleUnwrapKeyResponse);

  // SubtleSign signs a message using a key stored in the vault.
  rpc SubtleSign(SubtleSignRequest) returns (SubtleSignResponse);

  // SubtleVerify verifies the signature of a message using a key stored in the vault.
  rpc SubtleVerify(SubtleVerifyRequest) returns (SubtleVerifyResponse);

  // Encrypt encrypts a message using the Dapr encryption scheme and a key stored in the vault.
  rpc Encrypt(stream EncryptRequest) returns (stream EncryptResponse);

  // Decrypt decrypts a message using the Dapr encryption scheme and a key stored in the vault.
  rpc Decrypt(stream DecryptRequest) returns (stream DecryptResponse);
}

// rpc SubtleGetKey(SubtleGetKeyRequest) returns (SubtleGetKeyResponse);

// SubtleGetKeyRequest is the request object for SubtleGetKey.
message SubtleGetKeyRequest {
  enum KeyFormat {
    // PEM (SPKI) (default)
    PEM = 0;
    // JSON (JSON Web Key) as string
    JSON = 1;
  }
 
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key to use in the key vault
  string name = 2;
  // Response format
  KeyFormat format = 3;
}

// SubtleGetKeyResponse is the response for SubtleGetKey.
message SubtleGetKeyResponse {
  // Name (or name/version) of the key.
  // This is returned as response too in case there is a version.
  string name = 1;
  // Public key, encoded in the requested format
  string public_key = 2 [json_name="publicKey"];
}

// rpc SubtleEncrypt(SubtleEncryptRequest) returns (SubtleEncryptResponse);

// SubtleEncryptRequest is the request for SubtleEncrypt.
message SubtleEncryptRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Message to encrypt.
  bytes plaintext = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 6 [json_name="associatedData"];
}

// SubtleEncryptResponse is the response for SubtleEncrypt.
message SubtleEncryptResponse {
  // Encrypted ciphertext.
  bytes ciphertext = 1;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 2;
}

// rpc SubtleDecrypt(SubtleDecryptRequest) returns (SubtleDecryptResponse);

// SubtleDecryptRequest is the request for SubtleDecrypt.
message SubtleDecryptRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Message to decrypt.
  bytes ciphertext = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 6;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 7 [json_name="associatedData"];
}

// SubtleDecryptResponse is the response for SubtleDecrypt.
message SubtleDecryptResponse {
  // Decrypted plaintext.
  bytes plaintext = 1;
}

// rpc SubtleWrapKey(SubtleWrapKeyRequest) returns (SubtleWrapKeyResponse);

// SubtleWrapKeyRequest is the request for SubtleWrapKey.
message SubtleWrapKeyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Key to wrap
  bytes plaintext_key = 2 [json_name="plaintextKey"];
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 6 [json_name="associatedData"];
}

// SubtleWrapKeyResponse is the response for SubtleWrapKey.
message SubtleWrapKeyResponse {
  // Wrapped key.
  bytes wrapped_key = 1 [json_name="wrappedKey"];
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 2;
}

// rpc SubtleUnwrapKey(SubtleUnwrapKeyRequest) returns (SubtleUnwrapKeyResponse);

// SubtleUnwrapKeyRequest is the request for SubtleUnwrapKey.
message SubtleUnwrapKeyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Wrapped key.
  bytes wrapped_key = 2 [json_name="wrappedKey"];
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 6;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 7 [json_name="associatedData"];
}

// SubtleUnwrapKeyResponse is the response for SubtleUnwrapKey.
message SubtleUnwrapKeyResponse {
  // Key in plaintext
  bytes plaintext_key = 1 [json_name="plaintextKey"];
}

// rpc SubtleSign(SubtleSignRequest) returns (SubtleSignResponse);

// SubtleSignRequest is the request for SubtleSign.
message SubtleSignRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Digest to sign.
  bytes digest = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
}

// SubtleSignResponse is the response for SubtleSign.
message SubtleSignResponse {
  // The signature that was computed
  bytes signature = 1;
}

// rpc SubtleVerify(SubtleVerifyRequest) returns (SubtleVerifyResponse);

// SubtleVerifyRequest is the request for SubtleVerify.
message SubtleVerifyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Digest of the message.
  bytes digest = 2;
  // Signature to verify.
  bytes signature = 3;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 4;
  // Name (or name/version) of the key.
  string key = 5;
}

// SubtleVerifyResponse is the response for SubtleVerify.
message SubtleVerifyResponse {
  // True if the signature is valid.
  bool valid = 1;
}

// rpc Encrypt(stream EncryptRequest) returns (stream EncryptResponse);

message EncryptRequest {
  // Request details. Must be present in the first message only.
  EncryptRequestOptions options = 1;
  // Chunk of data of arbitrary size.
  common.v1.StreamPayload payload = 2;
}

message EncryptRequestOptions {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key.
  string key = 2;
  // Force algorithm to use: "aes-gcm" or "chacha20-poly1305" (optional)
  string algorithm = 3;
}

message EncryptResponse {
  // Chunk of data.
  common.v1.StreamPayload payload = 1;
}

// rpc Decrypt(stream DecryptRequest) returns (stream DecryptResponse);

message DecryptRequest {
  // Request details. Must be present in the first message only.
  DecryptRequestOptions options = 1;
  // Chunk of data of arbitrary size.
  common.v1.StreamPayload payload = 2;
}

message DecryptRequestOptions {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key.
  string key = 2;
}

message DecryptResponse {
  // Chunk of data.
  common.v1.StreamPayload payload = 1;
}

For the common.v1.StreamPayload message, see https://github.com/dapr/dapr/pull/5170

The Encrypt and Decrypt methods are stream-based. Dapr will read from the client until it has sufficient data, and will then send back the encrypted/decrypted data to the client. Clients must thus both send data to the RPC and listen for incoming messages. SDKs can offer to consumer methods to read the data as a stream (e.g. in Go, they accept an io.Reader and return an io.Reader)

HTTP APIs

The HTTP APIs are developed in a way that is the exact "port" of the gRPC "subtle" APIs, and the contents of the request and response bodies match exactly the fields in the gRPC APIs.

Methods are:

• /v1.0/subtlecrypto/getkey -> SubtleGetKey
• /v1.0/subtlecrypto/encrypt -> SubtleEncrypt
• /v1.0/subtlecrypto/decrypt -> SubtleDecrypt
• /v1.0/subtlecrypto/wrapkey -> SubtleWrapKey
• /v1.0/subtlecrypto/unwrapkey -> SubtleUnwrapKey
• /v1.0/subtlecrypto/sign -> SubtleSign
• /v1.0/subtlecrypto/verify -> SubtleVerify

Currently, higher-level APIs are not offered via HTTP. This is because it's not possible, using HTTP, to have a stream where Dapr sends back data to the client before the client's request is complete. Without that, we'd have to buffer the entire data in-memory in the runtime, which would be not practical when encrypting very large files.

[olitomlinson]

Great proposal!

I'm struggling to get my head round the interoperation between the sidecar and the KeyVault / HSM, so maybe a sequence diagram might help aid my understanding of how it all hangs together :)

[ItalyPaleAle]

@olitomlinson does this help?

[olitomlinson]

@ItalyPaleAle

It really does help, I'm a visual learner, so many thanks for taking the time to provide the diagram.

1. My understanding is that the high-level encrypt/decrypt is implementing client-side/envelope encryption, where the encryption of payloads happens in the dapr runtime, and NOT in the KeyVault / KSM? Have I got that right? (I'm only asking because there was talk previously in this proposal that mentioned client-side/envelope encryption explicitly, but it is not mentioned in the latest iteration of the proposal)

---

2. There was previous discussion around allowing key names to be dynamic :

Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)

Will this make it into the proposal? I can definitely see a use-case where our KMS stores a key-per-tenant, and being able to provide this programmatically at runtime when invoking the high-level APIs would be fantastic.

There is a similar thread here where we talk of introducing the notion of a metadata.tenant argument passed onto the State Management building block API surface to introduce a variable on each invoke which is propagated to the component configuration

---

Currently, higher-level APIs are not offered via HTTP. This is because it's not possible, using HTTP, to have a stream where Dapr sends back data to the client before the client's request is complete. Without that, we'd have to buffer the entire data in-memory in the runtime, which would be not practical when encrypting very large files.

So I'm reading this as you're trying to protect developers from themselves, and channeling them to use gRPC just incase they ever need to encrypt large files? Alternatively, could you allow HTTP but set a max request size soft-limit to some upper bound that you feel comfortable allowing? Rationale : HTTP is incredibly accessible for developers, and many do not use gRPC, do we want to alienate those developers who are not comfortable with gRPC?