RFC: Support for Local Attestation (LA-TLS) between Enclaves

[vijaydhanraj]

Description of the problem

One of the cloud use cases is to modularize secure applications running in a system into different enclaves. This is to minimize the TCB and thereby reduce loss if some parts of the code are compromised. For example, one enclave may receive some personal user data and may interact with another enclave to service the user request. Due to the confidentiality nature of the data, both enclaves need to attest to each other before sharing data. But currently, Gramine supports local attestation only between a parent process and its forked child.

Creating this issue to identify potential pitfalls with this request and come up with a solid design to enable this feature.

[mkow]

This is to minimize the TCB and thereby reduce loss if some parts of the code are compromised. [...]

This whole paragraph seems to be based on incorrect assumptions - in Gramine all processes trust each other, if you compromise one of them then all of them are compromised, and this assumption is rather impossible in practice to remove (Gramine is +/- a distributed OS, making everything distrusting each other would make the design super hard, if not impossible).

Knowing parts of the original request, I think you wanted to actually write about local attestation between different Gramine instances, not inside one? If so, then that idea seemed insecure, because the connections between enclaves could be swapped (i.e. the specific "network" of enclaves is not attested), and had quite huge practical problems like breaking the cycles (both enclaves can't hardcode each other's hashes at the same time).

[DL8]

If so, then that idea seemed insecure, because the connections between enclaves could be swapped (i.e. the specific "network" of enclaves is not attested), and had quite huge practical problems like breaking the cycles (both enclaves can't hardcode each other's hashes at the same time).

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave (the first is derived from the signing public key, the others are signed in SIGSTRUCT). This will require some preliminary work to figure out the other enclave's MRENCLAVE, but at least the circular dependency is eliminated.

Support for Local Attestation (LA-TLS)

I don't think TLS is really needed here: using EREPORTs, we have a mechanism to authenticate both enclaves and have an integrity-protected channel, so the overhead introduced by TLS handshake is not really needed. Also, there is not much need to agree on a cipher suite. Instead, the report data can be used to perform a key exchange using some SIGMA protocol.

[vijaydhanraj]

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER

Yes, agree and in addition, maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

I don't think TLS is really needed here

The reason for having a TLS channel is to securely share a master key between the enclaves used for pipes' encryption as well as passing any other secrets/encryption key to the other enclave.

[DL8]

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER

Yes, agree and in addition, maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

I don't know the details of the protocol in case of fork, but it may be reusable.

I don't think TLS is really needed here

The reason for having a TLS channel is to securely share a master key between the enclaves used for pipes' encryption as well as passing any other secrets/encryption key to the other enclave.

We do want to establish a secure channel, but TLS is an overkill in this specific case, because by using EREPORT we have identity and integrity guarantees.

[mkow]

We do want to establish a secure channel, but TLS is an overkill in this specific case

I agree, and it's similar with other places where we use TLS in Gramine, but AFAIR we picked TLS because we couldn't find a good library implementing encrypted + integrity-checked byte streams with preshared keys and we didn't want to write a custom one.

you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave

@DL8: But how you then actually differentiate between one of your enclaves? You need to know who you're talking to, to verify whether the untrusted host didn't shuffle connections between your enclaves (i.e. connected your nginx process to the database instead of PHP process and now they're exchanging some garbage as trusted data). My point is rather high-level: you need to actually know the application you want to connect to, you can't leave this choice to the untrusted host, even if it's limited to the apps from that specific deployment.

[...] maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

@vijaydhanraj: I don't see how it's relevant anyhow here.

[vijaydhanraj]

I don't see how it's relevant anyhow here.

I wanted to add a unique per enclave element in the report (report data) that could be cross verified by the other party. With just MRSigner we don't have this uniqueness between the enclaves signed by the same author.

[DL8]

you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave

@DL8: But how you then actually differentiate between one of your enclaves? You need to know who you're talking to, to verify whether the untrusted host didn't shuffle connections between your enclaves (i.e. connected your nginx process to the database instead of PHP process and now they're exchanging some garbage as trusted data). My point is rather high-level: you need to actually know the application you want to connect to, you can't leave this choice to the untrusted host, even if it's limited to the apps from that specific deployment.

As enclave developer, you should define a different ISVPRODID (and possibly additional KSS fields) per enclave. In the example you provided, developer can define the following values for ISVPRODID:

• 0 - nginx server
• 1 - database
• 2 - PHP process

Since ISVPRODID is signed in SIGSTRUCT, it reflects the type of the enclave that was launched, except that it doesn't change across builds/versions.

[mkow]

except that it doesn't change across builds/versions

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs). But I guess you wanted to say "builds", not "builds/versions"?

[DL8]

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs).

For that purpose the ISVSVN and CONFIGSVN fields are in the report. It is the enclaves' responsibility to decide whether or not to accept the communication. Once accepted, I would expect enclave developer to add version information/handshake

As for functional versions:

But I guess you wanted to say "builds", not "builds/versions"?

I meant "builds/versions". Once both enclaves attested each other including their security versions, they should negotiate the actual version as part of their protocol. It's possible to use product ID fields to indicate the software version, but in my opinion it's a bit of an abuse of the mechanism and is harder to maintain

[boryspoplawski]

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs).

For that purpose the ISVSVN and CONFIGSVN fields are in the report. It is the enclaves' responsibility to decide whether or not to accept the communication. Once accepted, I would expect enclave developer to add version information/handshake

As for functional versions:

But I guess you wanted to say "builds", not "builds/versions"?

I meant "builds/versions". Once both enclaves attested each other including their security versions, they should negotiate the actual version as part of their protocol. It's possible to use product ID fields to indicate the software version, but in my opinion it's a bit of an abuse of the mechanism and is harder to maintain

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

[vijaydhanraj]

So, for two enclaves to mutually attest each other signed by the same author, we need

1. MRsigner to establish evidence.
2. ISVPRODID to establish identity (to ensure we are not interacting with other enclaves signed by the same author). And to further check versions, enclaves can use ISVSVN and CONFIGSVN.

Would this be fair to say?

[boryspoplawski]

@vijaydhanraj I don't think you need 2 at all - after 1 you have a secure channel and can exchange any identity information via it. You can trust the other endpoint of that channel, because it's all the same MRSIGNER. Since there was some assumption you need to exchange some information anyway (as @DL8 said), I see no point in 2.

[DL8]

@vijaydhanraj I don't think you need 2 at all - after 1 you have a secure channel and can exchange any identity information via it. You can trust the other endpoint of that channel, because it's all the same MRSIGNER. Since there was some assumption you need to exchange some information anyway (as @DL8 said), I see no point in 2.

I think you misunderstood me: my point from my comment above was that attestation guarantees only security version of the enclaves. Actual version negotiation, if needed, is part of the application's protocol and is out of scope for attestation.

Regarding the fields to use, MRSIGNER alone is not enough: it only identifies who signed the enclave, but there is absolutely no guarantee about the actual enclave's identity. The same key can be used to sign multiple enclaves, which may be entirely different and unrelated.

So, for two enclaves to mutually attest each other signed by the same author, we need

1. MRsigner to establish evidence.
2. ISVPRODID to establish identity (to ensure we are not interacting with other enclaves signed by the same author). And to further check versions, enclaves can use ISVSVN and CONFIGSVN.

Would this be fair to say?

I thought about it again, and I am not sure we can force any rules whether or not the report should be accepted. For example, if we force both enclaves to have the same MRSIGNER, it will block a developer that chose to trust 3rd party enclaves signed by selected other MRSIGNERs. To be pragmatic, I would expect local attestation to get a report verification callback, which will be invoked after the MAC verification. I would expect its signature to be something like this: bool verify_report_contents_callback(const sgx_report_t *report, void *user_data);

The enclave developer is expected implement the logic that decides whether or not to trust the other enclave. I would expect a reasonable enclave to verify the following:

1. MRSIGNER: the identity of the signer
2. ISVPRODID and ISVSVN: the identity of the enclave and its security version
3. Additional KSS fields if needed and supported

[vijaydhanraj]

I thought about it again, and I am not sure we can force any rules whether or not the report should be accepted. For example, if we force both enclaves to have the same MRSIGNER, it will block a developer that chose to trust 3rd party enclaves signed by selected other MRSIGNERs.

Thanks @DL8 but the point is to avoid trusted third parties (TTPs) and get attested locally. If we were to have any such TTPs, then one can simply use remote attestation, right?

[BFuhry]

Note that my answer is based on the following assumptions:

1. Enclave A with functionality AF wants to attest enclave B with functionality BF
2. Enclave A fully trusts the developer of enclave B
3. Enclave A knows MRSIGNER of enclave B

If the best design we can come up with anyhow requires some negotiation step between the enclaves, I fully agree with the statement of @boryspoplawski:

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

Involving other fields like ISVPRODID, ISVSVN, additional KSS fields, etc. seem not necessary in this case. Instead, Enclave A does local attestation with enclave B, checks the MRSIGNER, and establishes a secure channel. Afterwards, enclave B can just "say" to enclave A over the secure channel: I provide functionality BF. Even better, enclave B can say: I provide functionality BF in version BV. As a result, A would be sure that it talks to the enclave with the desired functionality and it could decide if the version is recent enough or not. Obviously, the same process can be done in the other direction for mutual trust.

Theoretically, B could lie about its functionality and version. However, this would contradict assumption 2. Furthermore, without assumption 2, ISVPRODID, ISVSVN, additional KSS fields, etc. would also not help as the developer of enclave B could lie in these fields as well.

An additional negotiation step might not even be necessary if BF and BV can be part of the report that B sends to A.

[vijaydhanraj]

Yes, true this Enclave A fully trusts the developer of enclave B this is critical for attestation using MRsigner.

@boryspoplawski @mkow @monavij But in cases where such trust cannot be expected, can we have an encrypted directory specified in the manifest between two enclaves that want to locally attest each other? Then both enclaves can share some randomly generated unique ID in this directory which can be compared with the report that is exchanged between the enclaves.

[DL8]

Thanks @DL8 but the point is to avoid trusted third parties (TTPs) and get attested locally. If we were to have any such TTPs, then one can simply use remote attestation, right?

If a another signer is trusted, I would expect it to be known at build time to have its MRSIGNER embedded it in the code. Therefore, a trusted third party (sort of a certificate authority) is not required.

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

Involving other fields like ISVPRODID, ISVSVN, additional KSS fields, etc. seem not necessary in this case. Instead, Enclave A does local attestation with enclave B, checks the MRSIGNER, and establishes a secure channel. Afterwards, enclave B can just "say" to enclave A over the secure channel: I provide functionality BF. Even better, enclave B can say: I provide functionality BF in version BV. As a result, A would be sure that it talks to the enclave with the desired functionality and it could decide if the version is recent enough or not. Obviously, the same process can be done in the other direction for mutual trust.

While enclave may choose to use only MRSIGNER to verify the other enclave, I see several problems with it:

1. Not checking ISVSVN (possibly CONFIGSVN) is a security bug: if security bugs are found in the enclave, the SVN is expected to be incremented once those were fixed. Therefore, if not checked, B might accept a vulnerable version of A
2. ISVPRODID, ISVEXTPRODID and ISVFAMILYID are already part of the report. Not considering them may lead to a maintenance headache: there may be several entirely different and unrelated enclaves signed by the same signer. Not checking those fields will force the developer to implement another layer of authentication for all enclaves, to determine if we're talking to B and not C
3. I find it a questionable practice to continue with the key exchange when we can already know that it's going to be rejected

Theoretically, B could lie about its functionality and version. However, this would contradict assumption 2. Furthermore, without assumption 2, ISVPRODID, ISVSVN, additional KSS fields, etc. would also not help as the developer of enclave B could lie in these fields as well.

ISVSVN, ISVPRODID and extended product ID fields are signed in SIGSTRUCT. Therefore, B lying about its functionality and security version is possible in one of the following cases:

1. The private key was stolen
2. A software bug

An additional negotiation step might not even be necessary if BF and BV can be part of the report that B sends to A.

That's basically my argument: we don't need an additional negotiation step because we already have relevant fields in the report.

Yes, true this Enclave A fully trusts the developer of enclave B this is critical for attestation using MRsigner.

This is up to the developer to decide. It can be in build time or in any other way.

[monavij]

We are mixing multiple things here. So let me try and clarify

1. With remote attestation you can verify that you are running on a good up to date Intel system and running in an enclave
2. With local attestation you can just verify that you are running in an enclave on the same machine (it might be out of date)
3. In both cases we need a Trusted party that can verify your MRENCLAVE. That can be a trusted third party or you can be your own trusted party. As long as you know who you are going to trust you can make sense out of MRENCLAVE value.

With this premise we can now build a solution that can enable encrypted unix domain sockets between two enclaves running on the same machine.

1. We already know we can use MRENCLAVE as the ID of the enclave. So we just create a signed white list of MRENCAVE values and include the public key of the signer in your manifest.
2. We could put this public key in KSS as well, in case you have different trusted parties on different machines, your MRENCLAVE value remains the same.
3. If there are just two enclaves then they can also include each others MRENCLAve value in KSS field.

[DL8]

We are mixing multiple things here. So let me try and clarify

1. With remote attestation you can verify that you are running on a good up to date Intel system and running in an enclave
2. With local attestation you can just verify that you are running in an enclave on the same machine (it might be out of date)
3. In both cases we need a Trusted party that can verify your MRENCLAVE. That can be a trusted third party or you can be your own trusted party. As long as you know who you are going to trust you can make sense out of MRENCLAVE value.

I don't agree with statement 3. As I see it, there are two main steps in the attestation:

1. Integrity check: enclave verifies the integrity of the report (EGETKEY and MAC). If this succeeded, we know the report came from an enclave running on the same platform, and that its contents indeed reflect its identity
2. Content check: enclave decides whether or not to accept the report. A trusted third party is not required: for example, the enclave may decide based on hard-coded expectations from MRSIGNER, ISVPRODID and ISVSVN

With this premise we can now build a solution that can enable encrypted unix domain sockets between two enclaves running on the same machine.

1. We already know we can use MRENCLAVE as the ID of the enclave. So we just create a signed white list of MRENCAVE values and include the public key of the signer in your manifest.
2. We could put this public key in KSS as well, in case you have different trusted parties on different machines, your MRENCLAVE value remains the same.
3. If there are just two enclaves then they can also include each others MRENCLAve value in KSS field.

Using MRENCLAVE will be very hard to maintain. For example, you would potentially have to update the list every time one enclave in the ecosystem is modified. For example, let's say we have at least two enclaves: A and B. If A was updated, you would have to add its new MRENCLAVE to the list and keep the old value to maintain compatibility. Moreover, you would have to maintain different lists per build configuration (e.g. debug vs. release).

Even if other fields are used instead, I still see some problems with the external list approach:

1. Revocation: how is an enclave revoked? a new signed list may be provided, but what prevents an attacker from giving the enclave a previous version of the list?
2. Flexibility: enclave may behave differently under certain circumstances. For example, let's say we have enclave A that communicates an in-memory key-value store B and a DB server C. If it wants to establish connection with B but gets a report from C, it would want to reject the connection. In the list approach this would not be possible. It might get even more complicated if different enclave configurations are supported via KSS
3. KSS fields are a scarce resource and may impact key derivation. Therefore, other potential use cases must be carefully examined before taking a decision to reserve part of them for Gramine usage
4. There is no guarantee that the other enclave uses Gramine

In conclusion, I think there are too many unknowns to get a proper decision. Therefore, once the report is verified, the decision must remain the responsibility of the application itself. Gramine can provide examples or built-in callbacks, but forcing a mechanism might not be the right thing to do.

[boryspoplawski]

For example, you would potentially have to update the list every time one enclave in the ecosystem is modified. For example, let's say we have at least two enclaves: A and B. If A was updated, you would have to add its new MRENCLAVE to the list and keep the old value to maintain compatibility. Moreover, you would have to maintain different lists per build configuration (e.g. debug vs. release).

The very same thing can be said about KSS or any other ID(s) based system. You need a way to differentiate between configs, it doesn't really matter what you threat as an ID.

Revocation: how is an enclave revoked? a new signed list may be provided, but what prevents an attacker from giving the enclave a previous version of the list?

The list would have to come from a trusted third part, presumably as a part of remote attestation (or via some secure channel established with remote attestation). This doesn't have to be a list, it can be any format you want it to be.

There is no guarantee that the other enclave uses Gramine

There is no point in attestation then, from Gramine point of view. App can do whatever it wants and attest the remote end, Gramine should not interfere (and hence there's no problem with this).

Generally speaking I firmly believe that Gramine should only act as and interface for HW features (just provide an easy to use API, like /dev/ files) and allow the app to do whatever it wants. Whether it uses MRENCLVE, KSS, anything, it does not matter - it's up to the app. We could provide some examples like we do with RA-TLS, but again, that would be only an example usage of the API. This wouldn't support the seamless encryption of UNIX domain sockets though (but I never believed this is a good idea).

[monavij]

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs? Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

[DL8]

The very same thing can be said about KSS or any other ID(s) based system. You need a way to differentiate between configs, it doesn't really matter what you threat as an ID.

Yes, but it's much easier to maintain and verify those fields, because they are not bound to change with each build. I would expect that for each enclave, developer defines a different constant for product ID in the manifest, and it will not change often (if at all). This can be set with sgx.isvprodid and other fields can be added for KSS support in the future. sgx.isvsvn will be used to define the security version of the enclave, so other parties can reject prior vulnerable versions.

There is no guarantee that the other enclave uses Gramine

There is no point in attestation then, from Gramine point of view. App can do whatever it wants and attest the remote end, Gramine should not interfere (and hence there's no problem with this).

Generally speaking I firmly believe that Gramine should only act as and interface for HW features (just provide an easy to use API, like /dev/ files) and allow the app to do whatever it wants. Whether it uses MRENCLVE, KSS, anything, it does not matter - it's up to the app. We could provide some examples like we do with RA-TLS, but again, that would be only an example usage of the API.

Sounds like we pretty much agree. As for usage, I would expect Gramine to provide a handle to verify the integrity of the report. The app is then expected to check the report's contents and decide.

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs?

Technically speaking, local attestation is done after both enclaves verified each others' reports. I believe that in most cases, there is enough data in the report for the enclave to reach a decision without using MRENCLAVE, so forcing a trusted party will increase the TCB without a real justification.

Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

How to establish a secure channel is an important topic that hasn't been discussed so far.

I have no opinion regarding this feature, but if implemented, I believe it should have (at least) the following in the handshake:

1. Communicating enclaves exchange reports with key exchange material (depending on the protocol)
2. In Gramine: each receiving enclave verify the integrity of the other's report
3. In the application: each receiving enclave verifies the content of the report and chooses whether or not to accept the connection

[boryspoplawski]

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs? Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

@monavij The problem is that Gramine itself would need to do the attestation, not only expose API/interfaces for the app to do that. /dev/ files wouldn't be enough, you would need some configuration of which enclaves can connect to which enclave on which UNIX domain socket i.e. an extensive configuration that would have to be tightly coupled with Gramine and would offer no flexibility, e.g. if we proceed with MRENCLAVE that would mean the config cannot be provided at build time, but have to arrive via secure channel with RA, if we go with MRSIGNER + KSS/ISV* fields then users would have to use them in their software, MRENCLAVE wouldn't be enough. Supporting both would make it even more complex.

How to establish a secure channel is an important topic that hasn't been discussed so far.

@DL8 For this we have code already, we do local attestation on fork() (but in our case child and parent have exactly same MRENCLAVE). It's just a matter of deciding who to trust/attest, the rest shouldn't be a problem

[monavij]

@boryspoplawski - Agree with you that for unix domain socket the onus is on Gramine to do the establishment of secure channel. Also we would like to make that mechanism flexible and let application decide how they want to establish trust in the other local enclaves. Not sure if that will make Gramine more complex or just more flexible. KSS is just another tool in our toolbox. We should add support for KSS in Gramine and allow applications to add whatever they want to use that field for. Gramine does not need to worry what they use it for. My main point is that we have mechanisms for being able to do secure Unix domain sockets.

[DL8]

@boryspoplawski - Agree with you that for unix domain socket the onus is on Gramine to do the establishment of secure channel. Also we would like to make that mechanism flexible and let application decide how they want to establish trust in the other local enclaves. Not sure if that will make Gramine more complex or just more flexible. KSS is just another tool in our toolbox. We should add support for KSS in Gramine and allow applications to add whatever they want to use that field for. Gramine does not need to worry what they use it for. My main point is that we have mechanisms for being able to do secure Unix domain sockets.

Sounds like an existing mechanism can be reused for this purpose, which may be the right thing to do. However, "raw attestation" interface may be needed as well, in case the other enclave doesn't use Gramine and another protocol is required. Regarding KSS, note that local attestation can be achieved without it, so it should not be a dependency. Also note that even if it is supported in the platform, it must be explicitly enabled in SECS attributes. The attributes are also part of the report, so the verifying enclave may make a decision based on KSS status.

[vijaydhanraj]

To summarize the scope:

1. For raw attestation:

   a. Gramine already exposes raw attestation interfaces via /dev/attestation. b. Local attestation between parent and forked child is already supported. c. For Local Attestation between 2 different enclaves running on the same host using Gramine LibOS:

   • No changes are needed in core Gramine LibOS.
   • Gramine can provide a reference library for local attestation similar to RA-TLS for remote attestation. (onus is one application to decide what they choose to trust)
     • MRsigner, ISVPRODID and ISVSVN should be sufficient.
     • The apps can use the library to exchange reports and verify mac.
     • Application can use the verified report to check PRODID to ensure it is communicating with the intended enclave.
     • If needed the application can also use reportdata to securely pass information between the enclaves. (From SDM: REPORTDATA is a 64-Byte data structure that is provided by the enclave and included in the REPORT. It can be used to securely pass information from the enclave to the target enclave)

   d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

   • Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

2. Seamless communication over Unix Domain Sockets between Parent and child enclave using Gramine. - Not sure if this is currently supported but have all things in place to enable it. @boryspoplawski can you please confirm?

3. Communication over Unix Domain Sockets between enclaves using Gramine from same signing authority. a. Pre-establish the two enclaves that may communicate over UDS. May need manifest changes something like,

   • Set the pathname [sun_path] for both enclaves to use.
   • Encrypted path where each enclave tells Gramine that is done doing LA of the other enclave. Gramine can check this before proceeding to establish a UDS channel.)

   b. Application should use the LA library from Gramine. c. Application verifies the identity of the other enclave. d. Application sets up a secure channel and exchanges an encryption wrap key. d. Application writes to the encrypted path that it is done with LA and trusts the other enclave. e. Gramine checks the encrypted path and sets up UDS communication.

   Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

4. Communication over Unix Domain Sockets between enclaves using Gramine from different signing authority. - Not supported. Need same MRsigner

5. Communication over Unix Domain Sockets between Gramine and other LibOSes - Not supported for now. Need some standardization in the near future)

Thoughts/comments, please.

[DL8]

To summarize the scope:

1. For raw attestation: a. Gramine already exposes raw attestation interfaces via /dev/attestation.

What do you mean by "raw attestation"? As I understand it, it's the use case where the app uses Gramine only to verify a given report's integrity. I would expect Gramine to provide higher level APIs for secure channel establishment, such that if both enclaves use Gramine, these can be used and Gramine will take care of the key exchange details.

c. For Local Attestation between 2 different enclaves running on the same host using Gramine LibOS:

• No changes are needed in core Gramine LibOS.

• Gramine can provide a reference library for local attestation similar to RA-TLS for remote attestation. (onus is one application to decide what they choose to trust)

  • MRsigner, ISVPRODID and ISVSVN should be sufficient.
  • The apps can use the library to exchange reports and verify mac.
  • Application can use the verified report to check PRODID to ensure it is communicating with the intended enclave.
  • If needed the application can also use reportdata to securely pass information between the enclaves. (From SDM: REPORTDATA is a 64-Byte data structure that is provided by the enclave and included in the REPORT. It can be used to securely pass information from the enclave to the target enclave)

Note that once MAC is verified, the application may use all of the report's fields to decide. Gramine should provide a callback to let the app verify the report's contents and act accordingly. If we provide examples, I would expect the most standard flow to verify ISVPRODID, ISVSVN and MRSIGNER.

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

3. Communication over Unix Domain Sockets between enclaves using Gramine from same signing authority. a. Pre-establish the two enclaves that may communicate over UDS. May need manifest changes something like,

   • Set the pathname [sun_path] for both enclaves to use.
   • Encrypted path where each enclave tells Gramine that is done doing LA of the other enclave. Gramine can check this before proceeding to establish a UDS channel.)

   b. Application should use the LA library from Gramine. c. Application verifies the identity of the other enclave. d. Application sets up a secure channel and exchanges an encryption wrap key. d. Application writes to the encrypted path that it is done with LA and trusts the other enclave. e. Gramine checks the encrypted path and sets up UDS communication.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest, because there are other fields to consider and the actual decision may depend on runtime circumstances.

4. Communication over Unix Domain Sockets between enclaves using Gramine from different signing authority. - Not supported. Need same MRsigner

Why limit this mechanism to the same signing authority? If the app provides a callback, we don't need this restriction.

[vijaydhanraj]

What do you mean by "raw attestation"?

Please see here the low-level interfaces exposed by Gramine. This is what I mean by raw attestation. https://gramine.readthedocs.io/en/stable/attestation.html#low-level-dev-attestation-interface And yes, the idea is to create a library that will verify the integrity of the report by checking the mac but whether to expose the entire report, or limit it to just a few fields, establish a secure channel and other APIs related to user interface need to be finalized with maintainers.

Note that once MAC is verified, the application may use all of the report's fields to decide. Gramine should provide a callback to let the app verify the report's contents and act accordingly. If we provide examples, I would expect the most standard flow to verify ISVPRODID, ISVSVN and MRSIGNER

Yes, it makes sense.

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

Please note the library is built based on low-level raw attestation interfaces (/dev/attestation) and when using with other LibOSes would need to be customized. This is what I meant by the comment.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

The key exchange will not use a UDS as this is what we want to set up after the local attestation and other verification steps.

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest

Agree this will be a little tightly coupled with Gramine but the idea behind the proposal is to make the end-user's life easier. With this proposal the user needs to set the reportdata, PRODID they expect in the manifest and Gramine can transparently check the integrity (from MAC), verify the MRsigner and other ISV* fields to ensure the other enclave is trusted and indeed the expected one. Gramine, can go ahead and then set up a UDS. But this imposes a restriction where the enclave identification is by MRSigner/ISV* fields.

because there are other fields to consider and the actual decision may depend on runtime circumstances.

Can you elaborate more on this?

Why limit this mechanism to the same signing authority? If the app provides a callback, we don't need this restriction.

We may need this if we were to go with the above option. If not, then yes, this restriction doesn't apply.

[DL8]

What do you mean by "raw attestation"?

Please see here the low-level interfaces exposed by Gramine. This is what I mean by raw attestation. https://gramine.readthedocs.io/en/stable/attestation.html#low-level-dev-attestation-interface And yes, the idea is to create a library that will verify the integrity of the report by checking the mac but whether to expose the entire report, or limit it to just a few fields, establish a secure channel and other APIs related to user interface need to be finalized with maintainers.

Thanks for the reference. For the low-level attestation API, I would expect a handle called /dev/attestation/verify_report to be added. The app is expected to use it as follows:

1. Write the contents of the report to verify
2. Read a single byte to get the MAC verification results:
   • 0 - fail
   • 1 - success

Note that for the scope of this handle I assume the app already has the report at hand (how to get it is out of scope). @mkow - I think this API could be useful added regardless of local attestation. Should it be a separate feature request?

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

Please note the library is built based on low-level raw attestation interfaces (/dev/attestation) and when using with other LibOSes would need to be customized. This is what I meant by the comment.

Yes, the non-Gramine-based enclave will have a different abstraction (or bare-bone Assembly), but this is out of scope for us.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

The key exchange will not use a UDS as this is what we want to set up after the local attestation and other verification steps.

A UDS is a communication channel between processes. Local attestation and key exchange are done on top of that to establish a secure channel over it (or any other IPC mechanism). It doesn't really matter whether the handshake is performed on the same insecure channel or a different one, as long as in the end both enclaves use the same scheme and keys. This is analogous to TLS, where a secure channel is established over an insecure TCP/IP connection (and then the same socket is used to continue the communication).

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest

Agree this will be a little tightly coupled with Gramine but the idea behind the proposal is to make the end-user's life easier. With this proposal the user needs to set the reportdata, PRODID they expect in the manifest and Gramine can transparently check the integrity (from MAC), verify the MRsigner and other ISV* fields to ensure the other enclave is trusted and indeed the expected one. Gramine, can go ahead and then set up a UDS. But this imposes a restriction where the enclave identification is by MRSigner/ISV* fields.

Well, it could make is slightly easier for the developer to predefine a set of accepted enclaves. However, it might make SVN increment slightly more of a headache: for example, let's say that enclave A is used by enclave B and C, all of them are signed by the same key (same MRSIGNER). If I understand correctly, the expected (minimal?) ISVSVN of A will have to be in the manifest of B and C along with its ISV* identifiers. Now, if a security update is applied to A, its ISVSVN is incremented in its manifest. With this solution, it will have to be incremented in B's and C's manifests, otherwise they may be able to communicate with a vulnerable version of A. On the other hand, the alternative may be that developer does that manually. On the technical side, note that REPORTDATA is the field that is used for the actual enclave communication, so it should not be added to the manifest (in this case I would expect Gramine to use it as part of the the key exchange protocol).

because there are other fields to consider and the actual decision may depend on runtime circumstances.

Can you elaborate more on this?

Let's say we have an enclave that is expected to communicate with a DB server (e.g. SQL server) and a key-value store (e.g. memcached), both running in enclave and all have the same MRSIGNER and proper ISVPRODID. If the enclave tries to connect to the DB server but gets a quote from the memcached server, it will make more sense to reject the connection, even though it is a trusted enclave. If it tries to connect to the DB and gets a DB, it will make sense to accept. I don't know if it's a real use case though, but it sounds possible.

[dimakuv]

Let me add more chaos to this issue :)

1. The name "LA-TLS" is a bit misleading, and I regret coining it. The original thought was "This looks very much like RA-TLS", but now I see significant differences:

   • RA-TLS builds on top of TLS code of the native app. In other words, native app already encrypts its communication channels with TLS, and RA-TLS simply "augments" the TLS handshake with additional SGX-specific verification steps.
   • LA-TLS builds on top of plaintext-communications code of the native app. Applications always use UDS (aka AF_UNIX) without TLS. Thus, LA-TLS must not only augment the TLS handshake, but actually wrap UDS communications in TLS first.
   • I don't have a better name for it now, so I'll continue calling it "LA-TLS".

2. There are two things to be implemented for LA-TLS:

   1. The wrapping of UDS sessions in TLS. This is similar to how we do it here (for pipes and one-Gramine-instance UDSes): https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_pipes.c
   2. The SGX LA-based TLS handshake. This is similar to how we do it here (during fork, for the parent and the child to establish a TLS channel): https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_process.c

3. I agree with @boryspoplawski and @mkow that the policies to decide which SGX enclaves to trust are hard to embed in core Gramine (either as some manifest options as some hard-coded policies). It was easy with forking because there we applied a strict policy of same-MRENCLAVE: https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_process.c#L111-L127

However, it is really hard to decide how to trust SGX enclaves in the general case. In RA-TLS, we have a special callback: https://github.com/gramineproject/gramine/blob/cf43bbca4c48456378b34621a1a271c6b3bd8148/tools/sgx/ra-tls/ra_tls.h#L67-L82

Maybe we could introduce a similar callback in LA-TLS, just like @DL8 suggested.

My main problem currently: How do we combine the "wrapping of UDS sessions in TLS" (must be implemented in core Gramine) and the "SGX LA-based TLS handshake" (should be implemented in a shared lib on top of core Gramine)? It feels like we'll need to introduce several callbacks from Gramine to the app to make LA-TLS workable. And this is not a good design.