CCF icon indicating copy to clipboard operation
CCF copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Error when running sandbox script in release enclave-type

Open RokasAidukas-Centric opened this issue 3 years ago • 6 comments

Hello,

I am currently attempting to run, a CCF node in SGX hardware mode (release enclave type).

To Reproduce Run /opt/ccf/bin/sandbox.sh --enclave-type release

Output:

Setting up Python environment...
No package/app specified. Defaulting to installed JS logging app
Python environment successfully setup
[11:44:42.947] Starting 1 CCF node...
[11:44:53.508] Error! Some nodes ran into issues:
[11:44:53.509] - Node [0]:
[error_driver2api sgx_enclave_common.cpp:261] Enclave not authorized to run, .e.g. provisioning enclave hosted in app without access rights to /dev/sgx_provision. You need add the user id to group sgx_prv or run the app as root.

[load_pce ../pce_wrapper.cpp:188] Error, call sgx_create_enclave for PCE fail [load_pce], SGXError:4004.


[11:44:53.509] Please raise a bug if the issue is unexpected: https://github.com/microsoft/CCF/issues/new?assignees=&labels=bug&template=bug_report.md&title=Unexpected%20error%20when%20running%20sandbox%20script

I tried adding the user to the sgx_prv group but with no help.

Run as root: /opt/ccf/bin/sandbox.sh --enclave-type release

Output:

Setting up Python environment...
No package/app specified. Defaulting to installed JS logging app
Python environment successfully setup
[11:33:37.152] Starting 1 CCF node...
Traceback (most recent call last):
  File "/opt/ccf/bin/infra/node.py", line 322, in _start
    self.remote.get_startup_files(self.common_dir)
  File "/opt/ccf/bin/infra/remote.py", line 910, in get_startup_files
    self.remote.get(self.pem, dst_path, timeout=REMOTE_STARTUP_TIMEOUT_S)
  File "/opt/ccf/bin/infra/remote.py", line 470, in get
    raise ValueError(path)
ValueError: /home/<NAME>/ccf_run/workspace/sandbox_0/0.pem

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/ccf/bin/start_network.py", line 190, in <module>
    run(args)
  File "/opt/ccf/bin/start_network.py", line 87, in run
    network.start_and_open(args)
  File "/opt/ccf/bin/infra/network.py", line 492, in start_and_open
    self.start(args, **kwargs)
  File "/opt/ccf/bin/infra/network.py", line 453, in start
    primary = self._start_all_nodes(args, **kwargs)
  File "/opt/ccf/bin/infra/network.py", line 336, in _start_all_nodes
    node.start(
  File "/opt/ccf/bin/infra/node.py", line 200, in starton my own hardware
    self._start(
  File "/opt/ccf/bin/infra/node.py", line 327, in _start
    raise RuntimeError(
RuntimeError: Error starting node 0

Expected behavior I expected it the node to start successfully in a similar manner to how it runs in virtual mode:

Run: /opt/ccf/bin/sandbox.sh --enclave-type virtual

Output:

Setting up Python environment...
No package/app specified. Defaulting to installed JS logging app
Python environment successfully setup
[13:09:18.380] Virtual mode enabled
[13:09:18.380] Starting 1 CCF node...
[13:09:22.968] Started CCF network with the following nodes:
[13:09:22.968]   Node [0] = https://127.0.0.1:8000
[13:09:22.969] You can now issue business transactions to the /opt/ccf/bin/../lib/libjs_generic application
[13:09:22.969] Loaded JS application: /opt/ccf/bin/../samples/logging/js
[13:09:22.969] Keys and certificates have been copied to the common folder: /home/<NAME>/ccf_run/workspace/sandbox_common
[13:09:22.969] See https://microsoft.github.io/CCF/main/use_apps/issue_commands.html for more information
[13:09:22.970] Press Ctrl+C to shutdown the network

Environment information Hardware:

OS:

  • Ubuntu 20.04 LTS
  • Kernel 5.6.0-1056-oem

Drivers & libs:

  • CCF - 2.0.7
  • Open Enclave - 0.18.2
  • Intel DCAP driver - 1.41
  • FLC support enabled, relevant output from the SGX verification script here:
Extended feature bits (EAX=07H, ECX=0H)
eax: 0 ebx: 2294e287 ecx: 40400004 edx: ac000400
sgx available: 1
sgx launch control: 1

Additional context I have also tried running this on an older CCF version (1.0.13) and on a docker image as suggested here, by using the following image:

  • mcr.microsoft.com/ccf/app/dev 2.0.7-sgx

with the container producing the following error:

Run: /opt/ccf/bin/sandbox.sh --enclave-type release

Output:

Setting up Python environment...
No package/app specified. Defaulting to installed JS logging app
Python environment successfully setup
[10:59:39.706] Starting 1 CCF node...
[10:59:45.286] Error! Some nodes ran into issues:
[10:59:45.287] - Node [0]:
[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-trunk-217/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:116] Failed to open Intel SGX device.

terminating with uncaught exception of type std::logic_error: Could not create enclave: OE_PLATFORM_ERROR


[10:59:45.287] Please raise a bug if the issue is unexpected: https://github.com/microsoft/CCF/issues/new?assignees=&labels=bug&template=bug_report.md&title=Unexpected%20error%20when%20running%20sandbox%20script

In a nutshell, I am unsuccessfully trying to launch a CCF release node on my own hardware and not a VM (that I have tried out successfully). Any help and guidance is much appreciated!

RokasAidukas-Centric avatar Aug 29 '22 11:08 RokasAidukas-Centric

@RokasAidukas-Centric Thank you for the detailed report. This is an error I haven't seen before, but have a a few debugging steps that should help track down the problem.

Firstly I just wanted to confirm something - are your original repro steps on bare metal hardware, or within Docker? I assume bare metal, since you later mention trying a Docker image in "Additional context". If this is all within Docker, then it may be as simple as exposing the SGX devices to the container, passing args like --device /dev/sgx_enclave:/dev/sgx_enclave --device /dev/sgx_provision:/dev/sgx_provision -v /dev/sgx:/dev/sgx when the container image is started. Also worth checking that sgx_provision and sgx_enclave devices are present under /dev on the hosting OS.

Failing that, it would be helpful to know if the attestation error is from OpenEnclave, or from the way CCF asks OpenEnclave for a quote. Could you try running some of the sample apps in OE, to see if they're able to fetch attestations or throw the same error.

eddyashton avatar Aug 30 '22 12:08 eddyashton

@eddyashton thanks for your reply. This is on bare metal hardware, apologies if the Docker mention posed some confusion, but wanted to add that seemingly any attempt at running CCF nodes in enclave release mode failed thus far.

That being said I am including some output for the OE samples:

helloworld

  • Built with CMake

Run: ~/OESamples/helloworld/build$ sudo make run

Output:

[ 10%] Generating helloworld_t.h, helloworld_t.c, helloworld_args.h
Generating edge routine, for the Open Enclave SDK.
Processing /home/<USER>/OESamples/helloworld/helloworld.edl.
Processing /opt/openenclave/include/openenclave/edl/syscall.edl.
Processing /opt/openenclave/include/openenclave/edl/epoll.edl.
Processing /opt/openenclave/include/openenclave/edl/fcntl.edl.
Processing /opt/openenclave/include/openenclave/edl/ioctl.edl.
Processing /opt/openenclave/include/openenclave/edl/poll.edl.
Processing /opt/openenclave/include/openenclave/edl/signal.edl.
Processing /opt/openenclave/include/openenclave/edl/socket.edl.
Processing /opt/openenclave/include/openenclave/edl/time.edl.
Processing /opt/openenclave/include/openenclave/edl/unistd.edl.
Processing /opt/openenclave/include/openenclave/edl/utsname.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/platform.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/attestation.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/cpu.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/debug.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/thread.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/switchless.edl.
Success.
Scanning dependencies of target enclave
[ 20%] Building C object enclave/CMakeFiles/enclave.dir/enc.c.o
[ 30%] Building C object enclave/CMakeFiles/enclave.dir/helloworld_t.c.o
[ 40%] Linking C executable enclave
[ 40%] Built target enclave
Scanning dependencies of target sign
[ 50%] Generating private.pem, public.pem
Generating RSA private key, 3072 bit long modulus (2 primes)
.......................................................++++
...............................................................................................................................................................................................................................................................++++*...............................................................................................................++++
e is 3 (0x03)
writing RSA key
[ 60%] Generating enclave/enclave.signed
Created /home/<USER>/OESamples/helloworld/build/enclave/enclave.signed
[ 60%] Built target sign
[ 70%] Generating helloworld_u.h, helloworld_u.c, helloworld_args.h
Generating edge routine, for the Open Enclave SDK.
Processing /home/<USER>/OESamples/helloworld/helloworld.edl.
Processing /opt/openenclave/include/openenclave/edl/syscall.edl.
Processing /opt/openenclave/include/openenclave/edl/epoll.edl.
Processing /opt/openenclave/include/openenclave/edl/fcntl.edl.
Processing /opt/openenclave/include/openenclave/edl/ioctl.edl.
Processing /opt/openenclave/include/openenclave/edl/poll.edl.
Processing /opt/openenclave/include/openenclave/edl/signal.edl.
Processing /opt/openenclave/include/openenclave/edl/socket.edl.
Processing /opt/openenclave/include/openenclave/edl/time.edl.
Processing /opt/openenclave/include/openenclave/edl/unistd.edl.
Processing /opt/openenclave/include/openenclave/edl/utsname.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/platform.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/attestation.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/cpu.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/debug.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/thread.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/switchless.edl.
Success.
Scanning dependencies of target helloworld_host
[ 80%] Building C object host/CMakeFiles/helloworld_host.dir/host.c.o
[ 90%] Building C object host/CMakeFiles/helloworld_host.dir/helloworld_u.c.o
[100%] Linking C executable helloworld_host
[100%] Built target helloworld_host
Scanning dependencies of target run
Hello world from the enclave
Enclave called into host to print: Hello World!
[100%] Built target run

file-encryptor

  • Built with CMake

Run: ~/OESamples/file-encryptor$ make run

Output:

[ 83%] Built target enclave
[100%] Built target sign
Scanning dependencies of target run
Host: enter main
Host: create enclave for image:/home/<USER>/OESamples/file-encryptor/build/enclave/enclave.signed
Host: encrypting file:/home/<USER>/OESamples/file-encryptor/testfile -> file:./out.encrypted
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/encryptor.cpp(30): ecall_dispatcher::initialize : encrypting request
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(270): prepare_encryption_header
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(62): generate_password_key
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(99): Key based on password successfully generated
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(284): produce a encryption key
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(115): generate_encryption_key:
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(144): Encryption key successfully generated: a 32 byte key (hex):  
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(297): generate a digest for the password
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(308): encrypt the encryption key with a psswd key
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(168): cipher_encryption_key: encrypting
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(204): ecall_dispatcher::cipher_encryption_key
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(324): Done with prepare_encryption_header successfully.
Host: leftover_bytes 10
Host: start encrypting
Host: Working the last block
Host: padded_byte_count 6
Host: leftover_bytes 10
Host: done  encrypting
Host: called close_encryptor
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/encryptor.cpp(116): ecall_dispatcher::close
Host: compared file:./out.encrypted to file:./out.decrypted
Host: two files are not equal
Host: /home/<USER>/OESamples/file-encryptor/testfile is NOT equal to ./out.decryptedas expected
Host: encryption was done successfully
Host: decrypting file:./out.encrypted to file:./out.decrypted
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/encryptor.cpp(30): ecall_dispatcher::initialize : decrypting request
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(62): generate_password_key
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(99): Key based on password successfully generated
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(168): cipher_encryption_key: decrypting
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/keys.cpp(204): ecall_dispatcher::cipher_encryption_key
Host: leftover_bytes 0
Host: start decrypting
Host: done  decrypting
Host: called close_encryptor
Enclave: /home/<USER>/OESamples/file-encryptor/enclave/mbedtls_src/encryptor.cpp(116): ecall_dispatcher::close
Host: compared file:./out.encrypted to file:./out.decrypted
Host: two files are equal
Host: /home/<USER>/OESamples/file-encryptor/testfile is equal to ./out.decrypted
Host: terminate the enclave
Host: Sample completed successfully.
[100%] Built target run

attestation

This one was more interesting, the first one to fail out of the samples I have run.

  • Built with CMake

Run: ~/OESamples/attestation/build$ make run

Output:

[ 23%] Built target public_key_a
[ 47%] Built target common
[ 61%] Built target enclave_b
[ 71%] Built target enclave_b_signed
[ 76%] Built target public_key_b
[ 90%] Built target enclave_a
[100%] Built target enclave_a_signed
[100%] Built target sign
Scanning dependencies of target runsgxlocal
Host: Creating two enclaves
Host: Enclave library /home/<USER>/OESamples/attestation/build/enclave_a/enclave_a.signed
Enclave1: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(80): mbedtls initialized.
Host: Enclave successfully created.
Host: Enclave library /home/<USER>/OESamples/attestation/build/enclave_b/enclave_b.signed
Enclave2: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(80): mbedtls initialized.
Host: Enclave successfully created.
Host: environment variable SGX_AESM_ADDR is not set


Host: ********** Attest enclave_a to enclave_b **********

Host: Requesting enclave_b format settings
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(79): get_enclave_format_settings
Host: Requesting enclave_a to generate a targeted evidence with an encryption key
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(133): get_evidence_with_public_key
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(94): oe_serialize_custom_claims
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(105): serialized custom claims buffer size: 121
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(126): generate_attestation_evidence succeeded.
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(179): get_evidence_with_public_key succeeded
Host: enclave_a's  public key: 
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqimvDsNzh+kIunCYK6Rh
dRw9h8Nc7sIJJcnwtZLwPMdsdPBDcmQGtEtKLKKKH/XAY7SIzBwaguaNWLRpwUc+
DXyeUOfUNad6H6RKZehMHp6YL67oPNjdv+Nz8bgydlGaNfSFRQFhvi8aqt4hEpUv
wj/Ys/U5m13/2ulsF5VixwjOO2TLTU9MPzHWpDgOtYbDc/0AtQHxYWEaNqeKrwnb
yNn08dDgqtRb1sqPiDTphtgrp0/iteJUhVG4ZA5KlUgZi1Ce+s++3bab+TwH+9vn
R1lGrPo9nTL5ih6ghk4zvehzmnhli0mEMKOsv55bwXjOYfRrRjXMHSZrTAZQ3QsS
MQIDAQAB
-----END PUBLIC KEY-----

Host: verify_evidence_and_set_public_key in enclave_b
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(205): oe_verify_evidence succeeded
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(304): oe_deserialize_custom_claims
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(318): custom claim 1(Event): Attestation sample
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(320): custom claim 2(Public key hash) hash check:
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(328): hash match
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(331): attestation succeeded
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(231): verify_evidence_and_set_public_key succeeded.


Host: ********** Attest enclave_b to enclave_a **********

Host: Requesting enclave_a format settings
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(79): get_enclave_format_settings
Host: Requesting enclave_b to generate a targeted evidence with an encryption key
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(133): get_evidence_with_public_key
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(94): oe_serialize_custom_claims
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(105): serialized custom claims buffer size: 121
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(126): generate_attestation_evidence succeeded.
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(179): get_evidence_with_public_key succeeded
Host: enclave_b's  public key: 
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAspnbxciq2UX9z9G7Eq2r
JI0RTwz027bWZ03p+PJTs7RMBv4slqbr63zcgXqMTBPKwKkUwDbXutPaUEsLRYZM
AjPW2oNIKPsDufSWaTRRiaTCZzEDjPH0pSMDmyzXaT6ut25UnDbjwiOBLf/C0KfU
S9KWhJnq/q7lVVtWwOKh45rXHhC+Va7v5CMON69kfhlW2AyVwoCFTsFSP3F9B1Gg
SvdsRVe4BFbsWAVNEbTJOY0Wa+F3DMcvj/+MXicduFTJAiOrPFsdwWCPMXy2qB9g
PcR16MSvUIYO4yjcT5z/SFHKih/OR7EgaaJ1OJv0v6gBRKzUohBiLdKfF02kTiZM
cQIDAQAB
-----END PUBLIC KEY-----

Host: verify_evidence_and_set_public_key in enclave_a
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(205): oe_verify_evidence succeeded
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(304): oe_deserialize_custom_claims
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(318): custom claim 1(Event): Attestation sample
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(320): custom claim 2(Public key hash) hash check:
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(328): hash match
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(331): attestation succeeded
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(231): verify_evidence_and_set_public_key succeeded.
Host: Requesting encrypted message from 1st enclave
Enclave1: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(167): Padding used: MBEDTLS_RSA_PKCS_V21 for OAEP or PSS
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(272): enclave: generate_encrypted_message: encrypted_data_size = 256
Host: Sending the encrypted message to 2nd enclave
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(302): Decrypted data: 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(325): Decrypted data matches with the enclave internal secret data: descryption validation succeeded

***
Host: Now both enclaves have attested each other.
They can start exchanging messages between them 
using asymmetric encryption with the public keys exchanged earlier
***

Host: Requesting encrypted message from 2nd enclave
Enclave2: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(167): Padding used: MBEDTLS_RSA_PKCS_V21 for OAEP or PSS
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(272): enclave: generate_encrypted_message: encrypted_data_size = 256
Sending encrypted message to 1st enclave=====
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(302): Decrypted data: 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(325): Decrypted data matches with the enclave internal secret data: descryption validation succeeded
Host: Success
Host: Terminating enclaves
Enclave1: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(94): mbedtls cleaned up.
Host: Enclave successfully terminated.
Enclave2: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(94): mbedtls cleaned up.
Host: Enclave successfully terminated.
Host:  succeeded 
[100%] Built target runsgxlocal
Scanning dependencies of target runsgxremote
Host: Creating two enclaves
Host: Enclave library /home/<USER>/OESamples/attestation/build/enclave_a/enclave_a.signed
Enclave1: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(80): mbedtls initialized.
Host: Enclave successfully created.
Host: Enclave library /home/<USER>/OESamples/attestation/build/enclave_b/enclave_b.signed
Enclave2: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(80): mbedtls initialized.
Host: Enclave successfully created.
Host: environment variable SGX_AESM_ADDR is not set


Host: ********** Attest enclave_a to enclave_b **********

Host: Requesting enclave_b format settings
Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(79): get_enclave_format_settings
Host: Requesting enclave_a to generate a targeted evidence with an encryption key
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(133): get_evidence_with_public_key
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(94): oe_serialize_custom_claims
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(105): serialized custom claims buffer size: 121
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
Enclave1: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(126): generate_attestation_evidence succeeded.
Enclave1: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(179): get_evidence_with_public_key succeeded
Host: enclave_a's  public key: 
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsdMSAQt99DSEwWDOxbig
QrJVdDcqOiQZN3L0YA/Vh0FfCkpDCiesZ9d5yCh3x0x6GDwxQ+M4MuemBjhSocKP
BkTYOe0rlvM+5iFF5N1tq8YHd+gmaeJKrP3kxu8Wvgbb+M0bYaE4z82DdDPnIaDb
xs66X0KVPK+3tmFgpVg4VPG1yFYrBT4yz9vzrMdFIgt0n0omYvqqmgw4BA0UHt8H
0wJtetc745lACUbOECavH4dQe5QkFbeWpjGl0T/alP7XUZYtaJRN55EDMf+kXkeQ
ZrDlFefx/DoNVJB51ulfgu+HO8O+PXRb4w93f+Fjl2RkqDOGJOItOv+8Cvga4FBa
CQIDAQAB
-----END PUBLIC KEY-----

Host: verify_evidence_and_set_public_key in enclave_b
2022-08-30T16:52:41+0300.306035Z [(H)ERROR] tid(0x7fdcc729d740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/sgxquoteprovider.c:oe_get_sgx_quote_verification_collateral:121]
2022-08-30T16:52:41+0300.306150Z [(H)ERROR] tid(0x7fdcc729d740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/ocalls/ocalls.c:oe_get_quote_verification_collateral_with_baseline_ocall:239]
Enclave2: ***/home/<USER>/OESamples/attestation/common/attestation.cpp(201): oe_verify_evidence failed (OE_QUOTE_PROVIDER_CALL_ERROR).

Enclave2: ***/home/<USER>/OESamples/attestation/common/dispatcher.cpp(221): verify_evidence_and_set_public_key failed.
Host: verify_evidence_and_set_public_key failed. OE_OK
Host: attestation failed with 1
Host: Terminating enclaves
Enclave1: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(94): mbedtls cleaned up.
Host: Enclave successfully terminated.
Enclave2: ***/home/<USER>/OESamples/attestation/common/crypto.cpp(94): mbedtls cleaned up.
Host: Enclave successfully terminated.
Host:  failed 
make[3]: *** [CMakeFiles/runsgxremote.dir/build.make:57: CMakeFiles/runsgxremote] Error 1
make[2]: *** [CMakeFiles/Makefile2:169: CMakeFiles/runsgxremote.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:232: CMakeFiles/run.dir/rule] Error 2
make: *** [Makefile:144: run] Error 2

attested_tls

Run failed here as well, looks like some issues related to Azure DCAP.

  • Built with CMake

Run: ~/OESamples/attested_tls/build$ make run

Output:

[ 31%] Generating tls_server_enc.signed, tls_server_enc_mrenclave.h
Created /home/<USER>/OESamples/attested_tls/build/server/enc/tls_server_enc.signed
[ 31%] Built target tls_server_sign_enc
[ 42%] Built target tls_server_host
[ 42%] Built target tls_server
Scanning dependencies of target tls_non_enc_client
[ 45%] Building CXX object non_enc_client/CMakeFiles/tls_non_enc_client.dir/cert_verify_config.cpp.o
[ 48%] Building CXX object non_enc_client/CMakeFiles/tls_non_enc_client.dir/__/common/verify_callback.cpp.o
[ 51%] Linking CXX executable tls_non_enc_client
[ 54%] Built target tls_non_enc_client
[ 65%] Built target tls_client_host
Scanning dependencies of target tls_client_enc
[ 68%] Building CXX object client/enc/CMakeFiles/tls_client_enc.dir/cert_verify_config.cpp.o
[ 71%] Building CXX object client/enc/CMakeFiles/tls_client_enc.dir/__/__/common/cert_verifier.cpp.o
[ 74%] Building CXX object client/enc/CMakeFiles/tls_client_enc.dir/__/__/common/identity_verifier.cpp.o
[ 77%] Linking CXX executable tls_client_enc
[ 94%] Built target tls_client_enc
[ 97%] Generating tls_client_enc.signed
Created /home/<USER>/OESamples/attested_tls/build/client/enc/tls_client_enc.signed
[ 97%] Built target tls_client_sign_enc
Scanning dependencies of target tls_client
[ 97%] Built target tls_client
Scanning dependencies of target run
[100%] Launch processes to establish an Attested TLS between an non-enclave TLS client and an TLS server running inside an enclave 
server port = 12341
Host: Creating an tls client enclave
Host: Enclave library /home/<USER>/OESamples/attested_tls/build/server/enc/tls_server_enc.signed
Host: Enclave successfully created.
Host: calling setup_tls_server
TLS server: Setup the listening TCP socket on SERVER_IP= [0.0.0.0] server_port = [12341]
TLS server: mbedtls_net_bind returned successfully. (listen_fd = 3)
TLS server: Seeding the random number generator (RNG)
TLS server: Generating the certificate and private key
public key used:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErcV+X0SU220S4pDYnivZsZ2I5VNA
dN7We3ASRoTeUbZx8Z3XCXWvS/fioR6YH3NA4y7SaWphATTkhJ2h1+buhw==
-----END PUBLIC KEY-----

Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
server name = [localhost]
server port = [12341]
Host: Creating two enclaves
Host: Enclave library /home/<USER>/OESamples/attested_tls/build/client/enc/tls_client_enc.signed
Host: Enclave successfully created.
Host: launch TLS client to initiate TLS connection
TLS client: 
Seeding the random number generator...
TLS client: Connecting to tcp: localhost/12341...
TLS client: Connected to server @localhost.12341
TLS client: Generating the certificate and private key
public key used:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8A+nEBJAreaZ9/26u949SNlcqRrX
9sZZ+dxcQac2wv6m2H1xaHke9MlpGE/cWQoaiZGHa1XEN70/g1vb+pNHXA==
-----END PUBLIC KEY-----

Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
TLS server: 
Setting up the SSL configuration....
TLS client: Setting up the SSL/TLS structure...
TLS client: mbedtls_ssl_config_defaults returned successfully
TLS client: Performing the SSL/TLS handshake...
TLS server: Waiting for a client connection request...
TLS client: Received TLS certificate
TLS client: cert_verify_callback with depth = 0
TLS client: crt->version = 3 certificate_buffer_size = 5276
2022-08-30T16:57:00+0300.280375Z [(H)ERROR] tid(0x7f9c4333e740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/sgxquoteprovider.c:oe_get_sgx_quote_verification_collateral:121]
2022-08-30T16:57:00+0300.280425Z [(H)ERROR] tid(0x7f9c4333e740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/ocalls/ocalls.c:oe_get_quote_verification_collateral_with_baseline_ocall:239]
TLS client: oe_verify_attestation_certificate_with_evidence_v2 failed with result = OE_QUOTE_PROVIDER_CALL_ERROR
TLS client: Failed
  ! mbedtls_ssl_handshake returned -0xffffffff

TLS client: Last error was: 1 - ERROR - Generic error
TLS server: failed
  ! mbedtls_ssl_handshake returned -0x7780
TLS server: server communication error -30592
TLS server: Last error was: -30592 - SSL - A fatal alert message was received from our peer

Host: Terminating enclaves
Host: Enclave successfully terminated.
Host:  succeeded 
Host: Enclave successfully terminated.
Host:  succeeded 
server port = 12345
Host: Creating an tls client enclave
Host: Enclave library /home/<USER>/OESamples/attested_tls/build/server/enc/tls_server_enc.signed
Host: Enclave successfully created.
Host: calling setup_tls_server
TLS server: Setup the listening TCP socket on SERVER_IP= [0.0.0.0] server_port = [12345]
TLS server: mbedtls_net_bind returned successfully. (listen_fd = 3)
TLS server: Seeding the random number generator (RNG)
TLS server: Generating the certificate and private key
public key used:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErcV+X0SU220S4pDYnivZsZ2I5VNA
dN7We3ASRoTeUbZx8Z3XCXWvS/fioR6YH3NA4y7SaWphATTkhJ2h1+buhw==
-----END PUBLIC KEY-----

Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'

StartingTLS client: 


TLS client: connected to localhost:12345
TLS server: 
Setting up the SSL configuration....
TLS server: Waiting for a client connection request...
TLS client: verify_callback called with preverify_ok=0
TLS client: self-signed certificated detected
TLS client: verify_callback called with preverify_ok=1
TLS client: certificate=0x23deda0 buffer=0x23e023b buffer moved by 5275 offset certificate_size=5275
 verifying certificate start 
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
2022-08-30T16:57:18+0300.563683Z [(H)ERROR] tid(0x7fdf2a7dff40) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/sgxquoteprovider.c:oe_get_sgx_quote_verification_collateral:121]
2022-08-30T16:57:18+0300.563745Z [(H)ERROR] tid(0x7fdf2a7dff40) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/common/sgx/collateral.c:oe_get_sgx_quote_verification_collateral_from_certs:190]
2022-08-30T16:57:18+0300.563763Z [(H)ERROR] tid(0x7fdf2a7dff40) | Failed to get certificate quote verification collateral information. OE_QUOTE_PROVIDER_CALL_ERROR (oe_result_t=OE_QUOTE_PROVIDER_CALL_ERROR) [/source/openenclave/common/sgx/endorsements.c:oe_get_sgx_endorsements:405]
2022-08-30T16:57:18+0300.564174Z [(H)ERROR] tid(0x7fdf2a7dff40) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/common/sgx/verifier.c:oe_sgx_verify_evidence:879]
2022-08-30T16:57:18+0300.564196Z [(H)ERROR] tid(0x7fdf2a7dff40) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/common/attest_plugin.c:oe_verify_evidence:438]
2022-08-30T16:57:18+0300.564213Z [(H)ERROR] tid(0x7fdf2a7dff40) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/common/attest_plugin.c:oe_verify_attestation_certificate_with_evidence_v2:617]
TLS client: oe_verify_attestation_certificate_with_evidence_v2 failed with result=OE_QUOTE_PROVIDER_CALL_ERROR
TLS client: verifying SGX certificate extensions ... failed
TLS client: Error: Could not establish an SSL session ret2=-1 SSL_get_error()=1
TLS client:  failed
TLS server: failed
  ! mbedtls_ssl_handshake returned -0x7780
TLS server: server communication error -30592
TLS server: Last error was: -30592 - SS

Good idea trying to run some of the OE samples, does this help indicate any potential issues I may be facing?

Kind regards.

RokasAidukas-Centric avatar Aug 30 '22 14:08 RokasAidukas-Centric

@RokasAidukas-Centric Thanks for running those. The attestation test failure is the most interesting, since it's the most likely to be making the same calls that the CCF node makes that result in your initial failure. However since the error is different, it doesn't really help us narrow it down. Put another way, the fact that the initial attestation fetches in the OE sample passed, while the CCF node was unable to even contact the sgx device, suggests we have some additional permission issue.

A useful debugging step would be to see if you can launch a CCF node manually, without using the sandbox infrastructure. sandbox.sh is a wrapper around a Python script, which automates several steps (creating keys, creating configs, copying files) to set up a CCF network, but all those steps can also be done by hand. If you add the --verbose argument to the sandbox, it will print more info on what it is doing, and one of the final lines before the crash should show the invocation that it uses to launch cchost, something like:

| INFO     | infra.remote:start:484 - [127.0.0.1] cd /data/src/1.CCF/build/workspace/sandbox_0 && ./cchost --config ...

If you can copy that command and run it by hand, you will start a CCF node (a cchost process instance) with the files/config created by the sandbox. You may need to clear the snapshot or ledger directories for this to work - the error messages should make that clear.

The end result here should be either:

  • cchost can be launched manually, but not by sandbox.sh (which suggests a simple linux permissions/groups loss in the sandbox/infra stack)
  • cchost cannot even be launched manually, and throws the same error message about being unable to access /dev/sgx_provision (which suggests a deeper problem with the sgx install, or how we access it)

Out of interest, is running on your own hardware (rather than a VM) a firm requirement for your use case, or simply an experiment?

eddyashton avatar Sep 01 '22 12:09 eddyashton

@eddyashton Thanks for your insights, it gives me a better understanding on what goes on under the hood here. I have recreated what you suggested by running the cchost using the sandbox_0 configuration and I reached the second outcome that cchost was unable to run. I did need to perform some of the tasks that you mentioned, like removing the ledger folder. After that the err file did not produce any direct errors but out further provided the same error as before:

[ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'

Full error:

2022-09-05T07:38:49.490150Z        100 [info ] ../src/host/main.cpp:122             | CCF version: ccf-2.0.7
2022-09-05T07:38:49.490503Z        100 [info ] ../src/host/main.cpp:130             | Configuration file /home/<USER>/workspace/sandbox_common/0.config.json:
{
  "enclave": {
    "file": "./libjs_generic.enclave.so.signed",
    "type": "Release"
  },
  "network": {
    "node_to_node_interface": { "bind_address": "127.0.0.1:0" },
    "rpc_interfaces": {"primary_rpc_interface": {"bind_address": "127.0.0.1:8000", "endorsement": {"authority": "Service"}, "http_configuration": {"max_body_size": "1048576", "max_header_size": "16384", "max_headers_count": 256}, "max_open_sessions_hard": 1010, "max_open_sessions_soft": 1000, "published_address": "127.0.0.1:8000"}}
  },
  "node_certificate":
  {
    "subject_name": "CN=CCF Node",
    "subject_alt_names": [],
    "curve_id": "Secp384R1",
    "initial_validity_days": 90
  },
  "node_data_json_file": null,
  "service_data_json_file": null,
  "command": {
    "type": "Start",
    "service_certificate_file": "service_cert.pem", 
    "start":
    {
      "members": [{"certificate_file": "/home/<USER>/workspace/sandbox_common/member0_cert.pem", "data_json_file": null, "encryption_public_key_file": "/home/<USER>/workspace/sandbox_common/member0_enc_pubk.pem"}],
      "constitution_files": ["/home/<USER>/workspace/sandbox_common/actions.js", "/home/<USER>/workspace/sandbox_common/validate.js", "/home/<USER>/workspace/sandbox_common/resolve.js", "/home/<USER>/workspace/sandbox_common/apply.js"],
      "service_configuration":
      {
        "recovery_threshold": 0,
        "maximum_node_certificate_validity_days": 365,
        "maximum_service_certificate_validity_days": 365,
        "reconfiguration_type": "OneTransaction"
      },
      "initial_service_certificate_validity_days": 90
    },
    "join":
    {
      "retry_timeout": "1s",
      "target_rpc_address": ""
    },
    "recover": {
      "initial_service_certificate_validity_days": 90,
      "previous_service_identity_file": ""
    }
  },
  "ledger":
  {
    "directory": "0.ledger",
    "read_only_directories": [],
    "chunk_size": "5000000"
  },
  "snapshots":
  {
    "directory": "0.snapshots",
    "tx_count": 10000,
    "read_only_directory": null
  },
  "logging":
  {
    "host_level": "Info",
    "format": "Text"
  },
  "consensus":
  {
    "type": "CFT",
    "message_timeout": "100ms",
    "election_timeout": "4000ms"
  },
  "ledger_signatures":
  {
    "tx_count": 5000,
    "delay": "100ms"
  },
  "jwt":
  {
    "key_refresh_interval": "1800s"
  },
  "output_files": {
    "node_certificate_file": "0.pem",
    "pid_file": "node.pid",
    "node_to_node_address_file": "0.node_address",
    "rpc_addresses_file" : "0.rpc_addresses"
  },
  "tick_interval": "1ms",
  "slow_io_logging_threshold": "10ms",
  "client_connection_timeout": "2s",
  "node_client_interface": "127.100.0.0",
  "worker_threads": 0,
  "memory": {
    "circuit_size": "4MB",
    "max_msg_size": "16MB",
    "max_fragment_size": "64KB"
  }
}
2022-09-05T07:38:49.490584Z        100 [info ] ../src/host/main.cpp:161             | Recovery threshold unset. Defaulting to number of initial consortium members with a public encryption key (1).
2022-09-05T07:38:53.377403Z        100 [info ] ../src/host/ledger.h:1018            | Recovered ledger entries up to 0, committed to 0
2022-09-05T07:38:53.377507Z        100 [info ] ../src/host/lfs_file_handler.h:23    | Clearing contents from existing directory ".index"
2022-09-05T07:38:53.377849Z        100 [info ] ../src/host/node_connections.h:212   | Listening for node-to-node on 127.0.0.1:43441
2022-09-05T07:38:53.378056Z        100 [info ] ../src/host/rpc_connections.h:75     | Listening for RPCs on 127.0.0.1:8000
2022-09-05T07:38:53.378190Z        100 [info ] ../src/host/main.cpp:390             | Startup host time: 2022-09-05 10:38:53
2022-09-05T07:38:53.378587Z        100 [info ] ../src/host/main.cpp:438             | Creating new node: new network (with 1 initial member(s) and 1 member(s) required for recovery)
2022-09-05T07:38:53.378611Z        100 [info ] ../src/host/main.cpp:519             | Initialising enclave: enclave_create_node
2022-09-05T07:38:53.379292Z        100 [info ] ../src/host/enclave.h:186            | Padding config with 2 additional nulls
2022-09-05T07:38:53.496052Z        0   [info ] ../src/enclave/rpc_sessions.h:211    | Setting max open sessions on interface "primary_rpc_interface" (127.0.0.1:8000) to [1000, 1010] and endorsement authority to Service
2022-09-05T07:38:53.496131Z        0   [info ] ../src/node/node_state.h:1776        | Node TLS connections now accepted
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
2022-09-05T10:39:12+0300.680695Z [(H)ERROR] tid(0x7ff628a84740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/sgxquoteprovider.c:oe_get_sgx_quote_verification_collateral:121]
2022-09-05T10:39:12+0300.680754Z [(H)ERROR] tid(0x7ff628a84740) | :OE_QUOTE_PROVIDER_CALL_ERROR [/source/openenclave/host/sgx/ocalls/ocalls.c:oe_get_quote_verification_collateral_with_baseline_ocall:239]
2022-09-05T07:39:12.754797Z        0   [fail ] ../src/enclave/enclave.h:207         | Error starting node: Failed to get evidence: OE_QUOTE_PROVIDER_CALL_ERROR
2022-09-05T07:39:12.755115Z        100 [fail ] ../src/host/main.cpp:543             | An error occurred when creating CCF node: InternalError

To answer your last question, in my case it is a quite firm requirement, however even if it's a negative answer it is good for me as I am searching for proof whether using my own hardware is plausible in using CCF or not. I have doubts about it since the OE documentation stated that:

This step also installs the az-dcap-client package which is necessary for performing remote attestation in Azure. A general implementation for using Intel DCAP outside the Azure environment is coming soon. Found in here.

Do you think the current issue is more related to CCF or more related to OE?

Thanks!

RokasAidukas-Centric avatar Sep 05 '22 07:09 RokasAidukas-Centric

@RokasAidukas-Centric

To answer your last question, in my case it is a quite firm requirement, however even if it's a negative answer it is good for me as I am searching for proof whether using my own hardware is plausible in using CCF or not.

The answer to your question is that it is thought to be possible, because CCF relies on Open Enclave for all its interactions with SGX, including attestation, and Open Enclave supports running and using attestation outside Azure, as documented under: https://github.com/openenclave/openenclave/blob/master/docs/GettingStartedDocs/Contributors/NonAccMachineSGXLinuxGettingStarted.md

With that said, we have not tried this ourselves, and we aren't planning to at this time. Our ability to provide meaningful help is therefore limited, unfortunately.

achamayou avatar Sep 07 '22 16:09 achamayou

@achamayou thanks for your response, I will test this out and will report how it goes for me.

RokasAidukas-Centric avatar Sep 12 '22 05:09 RokasAidukas-Centric

@RokasAidukas-Centric please feel free to update the thread if possible.

achamayou avatar Feb 27 '23 23:02 achamayou

Hi @achamayou , I have been picking back these things up,

I recently successfully implemented a CCF-based backend on Azure and once I came back to do more research with v3 of CCF specifically on self hosted hardware, additionally I started using your v3 Docker images (in particular mcr.microsoft.com/ccf/app/dev:3.0.7-virtual). Virtual sandbox script worked as intended and I thought of trying to run a virtual cchost. And the virtual cchost failed here with some similar errors and it made me think - is a DNS name required to run even a virtual version of cchost?

I had similar issues in the past with the backend when I did not have a DNS name but that was done on cchost SGX.

Thanks!

RokasAidukas-Centric avatar Mar 03 '23 15:03 RokasAidukas-Centric

@RokasAidukas-Centric that's good to hear. No, a DNS is not necessary to run a cchost, you can absolutely just bind to IPs and leave it at that, that's what all our integration tests do for example.

Please do feel free to open a ticket with the details if you are still facing this or any other problem.

achamayou avatar Mar 08 '23 20:03 achamayou

Thanks, that's good to know, I will if any issues arise.

RokasAidukas-Centric avatar Mar 09 '23 07:03 RokasAidukas-Centric