gsc
gsc copied to clipboard
gramineproject
gramine-sgx-get-token fails to connect to aesmd from within a container
Description of the problem
This is an existing on-prem (Intel Nuc) system that has been used for graphene-sgx in the past. Our team is now looking to upgrade to gramine, and for the most part everything appears to be working correcly, however, when we attempt build and test container images, the test fails at runtime.
Any help or pointers are appreciated.
Steps to reproduce
- gsc commit id: 21b30f28e96d8ff83e1d7365ff84fea368d35bd3
- Run
./gsc build -d --insecure-args ubuntu18.04-bash test/ubuntu18.04-bash.manifest
./gsc sign-image ubuntu18.04-bash ~/.config/gramine/enclave-key.pem
docker run --privileged --device=/dev/isgx --device=/dev/gsgx -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-ubuntu18.04-bash -c ls
Expected results
Files are listed
Actual results
The following error is printed:
+ find /gramine/meson_build_output/lib -type d -path '*/site-packages'
+ export PYTHONPATH=:/gramine/meson_build_output/lib/python3.6/site-packages
+ PYTHONPATH=:/gramine/meson_build_output/lib/python3.6/site-packages
++ find /gramine/meson_build_output/lib -type d -path '*/pkgconfig'
+ export PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ '[' -z '' ']'
+ gramine-sgx-get-token --sig /entrypoint.sig --output /entrypoint.token
Attributes:
mr_enclave: 208579237095fcc1358557f23e5903c4bca6d9d36b495a319ba787b497b1e0bc
mr_signer: 89139782896bcfc6cdc46924b53fdc83b8886ea681230bc5bb77f584bacaf255
isv_prod_id: 0
isv_svn: 0
attr.flags: 0000000000000004
attr.xfrm: 0000000000000007
mask.flags: ffffffffffffffff
mask.xfrm: fffffffffff9ff1b
misc_select: 00000000
misc_mask: ffffffff
modulus: dd47372296ab46c9351e57564457be48...
exponent: 3
signature: e7327dff8d92bebf22f34a78c864ff4c...
date: 2022-01-06
Traceback (most recent call last):
File "/gramine/meson_build_output/bin/gramine-sgx-get-token", line 20, in <module>
main() # pylint: disable=no-value-for-parameter
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/gramine/meson_build_output/bin/gramine-sgx-get-token", line 16, in main
token = get_token(sig, verbose=verbose)
File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_get_token.py", line 152, in get_token
token = connect_aesmd(sig['enclave_hash'], sig['modulus'], sig['attribute_flags'], xfrms)
File "/gramine/meson_build_output/lib/python3.6/site-packages/graminelibos/sgx_get_token.py", line 89, in connect_aesmd
raise Exception(f'Failed. (Error Code = {ret_msg.ret.error})')
Exception: Failed. (Error Code = 6)
Additional information
SGX is installed, aesmd is running (see output from is-sgx-below) and our old graphene-sgx containers function correctly
is-sgx-available
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): false
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x5d80000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
Could you check (and paste here) the output of sudo service aesmd status on the host?
Also could you check the output of dmesg? It may also contain some interesting AESMD messages.
Also, can you show your config.yaml file?
Could you check (and paste here) the output of
sudo service aesmd statuson the host?
Thanks for the quick response. Here's the status:
sudo service aesmd status
● aesmd.service - Intel(R) Architectural Enclave Service Manager
Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-01-06 14:37:48 CST; 18h ago
Process: 4270 ExecStart=/opt/intel/libsgx-enclave-common/aesm/aesm_service (code=exited, status=0/SUCCESS)
Process: 4269 ExecStartPre=/bin/chmod 0755 /var/run/aesmd/ (code=exited, status=0/SUCCESS)
Process: 4268 ExecStartPre=/bin/chown -R aesmd:aesmd /var/run/aesmd/ (code=exited, status=0/SUCCESS)
Process: 4255 ExecStartPre=/bin/mkdir -p /var/run/aesmd/ (code=exited, status=0/SUCCESS)
Process: 4243 ExecStartPre=/opt/intel/libsgx-enclave-common/aesm/linksgx.sh (code=exited, status=0/SUCCESS)
Main PID: 4271 (aesm_service)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/aesmd.service
└─4271 /opt/intel/libsgx-enclave-common/aesm/aesm_service
Jan 06 14:37:48 ideanovalicensing systemd[1]: Starting Intel(R) Architectural Enclave Service Manager...
Jan 06 14:37:48 ideanovalicensing systemd[1]: Started Intel(R) Architectural Enclave Service Manager.
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]White List update requested
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]Platform Services initializing
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]Platform Services initialization failed due to DAL error
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: The server sock is 0x562731c2ee10
Jan 06 14:37:48 ideanovalicensing aesm_service[4271]: [ADMIN]White list update request successful for Version: 108
Also could you check the output of
dmesg? It may also contain some interesting AESMD messages.Also, can you show your
config.yamlfile?
Thanks for the suggestion. Here are my updates:
dmesg | grep -i aesmd
No output.
cat config.yaml
# Specify the OS distro. Currently tested distros are Ubuntu 18.04 and Ubuntu 20.04.
Distro: "ubuntu:18.04"
# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though
Gramine:
Repository: "https://github.com/gramineproject/gramine.git"
Branch: "master"
# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
# - legacy out-of-tree driver: use something like the below values, but adjust the branch name
# Repository: "https://github.com/01org/linux-sgx-driver.git"
# Branch: "sgx_driver_1.9"
#
# - DCAP out-of-tree driver: use something like the below values
# Repository: "https://github.com/intel/SGXDataCenterAttestationPrimitives.git"
# Branch: "DCAP_1.11 && cp -r driver/linux/* ."
#
# - DCAP in-kernel driver: use empty values like below
# Repository: ""
# Branch: ""
#
SGXDriver:
Repository: "https://github.com/01org/linux-sgx-driver.git"
Branch: "sgx_driver_2.5"
I just want to reiterate that our old graphene-sgx based containers are working as expected, so it seems aesmd is ok.
Have you tried normal Gramine? Like described in https://gramine.readthedocs.io/en/latest/quickstart.html (installing from an Ubuntu package) and in https://gramine.readthedocs.io/en/latest/devel/building.html (git-cloning the latest version).
Have you tried normal Gramine? Like described in https://gramine.readthedocs.io/en/latest/quickstart.html (installing from an Ubuntu package) and in https://gramine.readthedocs.io/en/latest/devel/building.html (git-cloning the latest version).
We have installed the package gramine-oot
We have also cloned the repo and builth the hello-world:
make SGX=1 SGX_SIGNER_KEY="$HOME"/.config/gramine/enclave-key.pem
cc -Wall -Wextra -O3 -c -o helloworld.o helloworld.c
cc helloworld.o -o helloworld
gramine-manifest \
-Dlog_level=error \
helloworld.manifest.template helloworld.manifest
gramine-sgx-sign \
--key /home/int_license/.config/gramine/enclave-key.pem \
--manifest helloworld.manifest \
--output helloworld.manifest.sgx
Attributes:
size: 0x10000000
thread_num: 4
isv_prod_id: 0
isv_svn: 0
attr.flags: 0x6
attr.xfrm: 0x3
misc_select: 0x0
Memory:
000000000fffd000-0000000010000000 [REG:R--] (manifest) measured
000000000ffdd000-000000000fffd000 [REG:RW-] (ssa) measured
000000000ffd9000-000000000ffdd000 [TCS:---] (tcs) measured
000000000ffd5000-000000000ffd9000 [REG:RW-] (tls) measured
000000000ff95000-000000000ffd5000 [REG:RW-] (stack) measured
000000000ff55000-000000000ff95000 [REG:RW-] (stack) measured
000000000ff15000-000000000ff55000 [REG:RW-] (stack) measured
000000000fed5000-000000000ff15000 [REG:RW-] (stack) measured
000000000fec5000-000000000fed5000 [REG:RW-] (sig_stack) measured
000000000feb5000-000000000fec5000 [REG:RW-] (sig_stack) measured
000000000fea5000-000000000feb5000 [REG:RW-] (sig_stack) measured
000000000fe95000-000000000fea5000 [REG:RW-] (sig_stack) measured
000000000ba71000-000000000babc000 [REG:R-X] (code) measured
000000000babc000-000000000fe95000 [REG:RW-] (data) measured
0000000000010000-000000000ba71000 [REG:RWX] (free)
Measurement:
a46f072128759ed4f0e935f82ed05dc4fd1be9f55e3c382b34a080cb5836ea9c
gramine-sgx-get-token \
--output helloworld.token --sig helloworld.sig
Attributes:
mr_enclave: a46f072128759ed4f0e935f82ed05dc4fd1be9f55e3c382b34a080cb5836ea9c
mr_signer: 89139782896bcfc6cdc46924b53fdc83b8886ea681230bc5bb77f584bacaf255
isv_prod_id: 0
isv_svn: 0
attr.flags: 0000000000000006
attr.xfrm: 0000000000000007
mask.flags: ffffffffffffffff
mask.xfrm: ffffffffffffff1b
misc_select: 00000000
misc_mask: ffffffff
modulus: dd47372296ab46c9351e57564457be48...
exponent: 3
signature: b51cbf1e4b95e50f92680c354d84276f...
date: 2022-01-07
Running as a regular user fails:
gramine-sgx helloworld
error: Permission denied on mapping enclave. You may need to set sysctl vm.mmap_min_addr to zero
error: ECREATE failed in allocating EPC memory: -1
error: Creating enclave failed: -1
error: load_enclave() failed with error -1
With sudo
sudo gramine-sgx helloworld
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
Hello, world
Thanks again for the help! Truly appreciated
error: Permission denied on mapping enclave. You may need to set sysctl vm.mmap_min_addr to zero
Gramine gives you this error message. It means that you should do something like this: sudo sysctl vm.mmap_min_addr=0. After this, you won't need to run Gramine as sudo.
Gramine gives you this error message. It means that you should do something like this:
sudo sysctl vm.mmap_min_addr=0. After this, you won't need to run Gramine as sudo.
Thanks @dimakuv. I can run gramine without sudo, however, the original gsc issue still persist:
gramine-sgx helloworld
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
Hello, world
Hmmm, I currently cannot think of anything else to check... Have you tried to build your GSC image with ./gsc build -d (debug mode)? This would give you more Gramine logs.
@dimakuv I appreciate your help. I am using the -d mode to build. My next thought is to add more debug statements in the python code to see if I can get more detail.
@jpelkonen Any updates? Were you able to resolve the issue? If yes, could you share what was wrong?