gramineproject
Error when import some module in python example
Description of the problem
Error happened when import some module in python examples. So, Any limitations when import module when run python program in gramine
Steps to reproduce
Just add the following code in helloworld.py
import uuid
Expected results
Hello World
Actual results
I run 3 times and got different result.
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
error: Unexpected memory fault occurred inside PAL (0x1eddf2d6)
[P1:T1:python3.8] error: failed writing checkpoint header to child process (ret = -32)
[P1:T1:python3.8] error: process creation failed
Hello World
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
Hello World
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
error: Unexpected memory fault occurred inside PAL (0x1edcc274)
Gramine commit hash
5b160c68260eaa57a9342762b33dbff1e3c8fb5a
Thanks @deliciousdish for the bug report. This looks like some corner case in Gramine, because something inside Gramine itself fails, based on this line of log:
error: Unexpected memory fault occurred inside PAL (0x1edcc274)
Someone from the Gramine team will need to debug this...
I checked the docs，it looks like all the module imported should be added to fs.mounts in python.manifest.template, right? The uuid module is at "/usr/lib/python3.8", so i add it to fs.mounts as the following, but the error is the same as above.
fs.mounts = [
{ path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
{ path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
{ path = "/usr", uri = "file:/usr" },
{ path = "{{ python.stdlib }}", uri = "file:{{ python.stdlib }}" },
{ path = "{{ python.distlib }}", uri = "file:{{ python.distlib }}" },
{ path = "/etc", uri = "file:/etc" },
{ path = "/usr/lib/python3.8", uri = "file:/usr/lib/python3.8"},
{ type = "tmpfs", path = "/tmp" },
]
The module installed by pip in my host is at the directory "$HOME/.local/lib/python3.8/site-packages"，When I add this path into fs.mounts and i got the following error message. How to add this path into the fs.mounts?
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
[P1:T1:] error: error mounting "/home/caiqishen/.local/lib/python3.8/site-packages" (chroot) under /home/caiqishen/.local/lib/python3.8/site-packages: -22
[P1:T1:] error: Error during libos_init() in init_mount (-22)
@deliciousdish Can you show your manifest.template file? I'm unsure why you get this error. Seems like some problem in the manifest.
I can't reproduce the second error, maybe i didn't add it right. But if i add "$HOME/.local/lib/python3.8/site-packages" into fs.mounts, will the "import" in python program find module in this directory or find it in "/usr/lib/python3/dist-packages"?
You may also check the kernel log:
sudo dmesg | grep ELDU
If it shows something like:
[ 2537.818570] ELDU returned 1073741837 (0x4000000d)
Then it's a kernel bug that causes "Unexpected memory fault occurred inside PAL". You'll need a different version of kernel.
No, these's nothing wrong in my kernel log. But I have solved this problem. I run gramine examples in a vm which is running a desktop computer with Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz. This cpu only supports SGX1 and has only 128M EPC. Then I set only 16M EPC for the VM. I think the too few EPC is the root cause of the error because i changed 16M to 64 M and no error happened.
@deliciousdish Thanks for root causing the problem. Very valuable information!
I'm still curious why Gramine fails in such a weird way... I would expect Gramine to terminate with some meaningful message, like "out of memory" or smth like this. Well, whatever.
I think we should diagnose this problem, trusted (enclave) part shouldn't fault like that (no matter what).
Yes, I agree that it would be good to diagnose this problem. I'm currently not sure how to reproduce this deployment though.
@deliciousdish Are you from Intel?
No. I tried to reproduce the error, but always got different error code.
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
error: Unexpected memory fault occurred inside PAL (0x1edbddec)
[P1:T1:python3.8] error: failed writing checkpoint header to child process (ret = -32)
[P1:T1:python3.8] error: process creation failed
Hello World
caiqishen@gramine-clean:~/gramine/CI-Examples/python$ gramine-sgx ./python scripts/helloworld.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- sgx.debug = true (this is a debug enclave)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
- sgx.allowed_files = [ ... ] (some files are passed through from untrusted host without verification)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
error: Unexpected memory fault occurred inside PAL (0x1edba2d6)
[P1:T1:python3.8] error: failed writing checkpoint header to child process (ret = -32)
[P1:T1:python3.8] error: process creation failed
Hello World
I can't upload logs maybe because i am behind a proxy. I paste some logs around error here.
debug: Adding pages to enclave: 0x1ede3000-0x1ede7000 [REG:RW-] (data) measured
debug: Adding pages to enclave: 0x1ede7000-0x1f1bb000 [REG:RW-] (bss) measured
debug: Adding pages to enclave: 0x10000-0x1ed9e000 [REG:RWX] (free)
debug: Added all pages to SGX enclave
debug: Enclave initializing:
debug: enclave id: 0x000000001ffff000
debug: mr_enclave: 985c7a462eacfa12e13e607e2fc3af7d97016f4248fe8962d2dbd94ac142d3fd
debug: Key exchange succeeded
debug: Received local report
debug: Verify report:
debug: cpu_svn: 06130204018000000000000000000000
debug: mr_enclave: 985c7a462eacfa12e13e607e2fc3af7d97016f4248fe8962d2dbd94ac142d3fd
debug: mr_signer: db03b4983246b938565563c73003abc5d14655514906db748c769ece7260bf2e
debug: attr.flags: 0000000000000007
debug: attr.xfrm: 0000000000000007
debug: isv_prod_id: 00
debug: isv_svn: 00
debug: report_data: 3305b169aa32f8cb6e95b533b3064b5a1936c6032115b06001e5b24f2c22e43b0000000000000000000000000000000000000000000000000000000000000000
debug: key_id: 13efeba5505cf1cdc90a98072e58818300000000000000000000000000000000
debug: mac: 6ba1ad70e21f5da27cb2c8aee98bb980
debug: Local attestation succeeded!
[P1:T1:python3.8] debug: allocating checkpoint store (size = 67108864, reserve = 33554432)
error: Unexpected memory fault occurred inside PAL (0x1edbddec)
[P1:T1:python3.8] debug: complete checkpointing data
[P1:T1:python3.8] debug: checkpoint of 58040 bytes created
[P1:T1:python3.8] error: failed writing checkpoint header to child process (ret = -32)
[P1:T1:python3.8] error: process creation failed
[P1:T1:python3.8] trace: ---- return from clone(...) = -32
[P1:T1:python3.8] trace: ---- close(7) = 0x0
[P1:T1:python3.8] trace: ---- close(6) = 0x0
[P1:T1:python3.8] trace: ---- close(3) = 0x0
[P1:T1:python3.8] trace: ---- close(4) = 0x0
[P1:T1:python3.8] trace: ---- close(5) = 0x0
[P1:T1:python3.8] trace: ---- stat("/scripts", 0x9b13820) = 0x0
[P1:T1:python3.8] trace: ---- stat("/usr/lib/python3.8", 0x9b13820) = 0x0
[P1:T1:python3.8] trace: ---- stat("/usr/lib/python3.8/lib-dynload", 0x9b13820) = 0x0
[P1:T1:python3.8] trace: ---- stat("/usr/lib/python3.8/lib-dynload/_uuid.cpython-38-x86_64-linux-gnu.so", 0x9b13500) = 0x0
[P1:T1:python3.8] trace: ---- openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_uuid.cpython-38-x86_64-linux-gnu.so", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:python3.8] trace: ---- read(3, 0x9b13188, 0x340) ...
[P1:T1:python3.8] trace: ---- return from read(...) = 0x340
[P1:T1:python3.8] trace: ---- newfstatat(3, "", 0x9b13020, 4096) = 0x0
[P1:T1:python3.8] trace: ---- mmap(0, 0x4130, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:python3.8] trace: ---- return from mmap(...) = 0x91b9000
[P1:T1:python3.8] trace: ---- mmap(0x91ba000, 0x1000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) ...
[P1:T1:python3.8] trace: ---- return from mmap(...) = 0x91ba000
[P1:T1:python3.8] trace: ---- mmap(0x91bb000, 0x1000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:python3.8] trace: ---- return from mmap(...) = 0x91bb000
It's indeed a corner case, because i run gramine in a vm(support sgx) with little epc(16M). If want to reproduce the error, you have to build qemu-sgx with commit id '5e0214cdeee17de949f2565f4429c15173179ae3', then run a vm with 16M EPC.