gramineproject
Failure to run app built with scala in OpenJDK8 on the latest gramine@86f0d3f
Description of the problem
App built with scala 2.12.15 failed with errors as
[P1:T1:] error: Error opening executable /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java: -2
[P1:T1:] error: Error during shim_init() in init_important_handles (-2)
Steps to reproduce
To ease reproduction, I've made this repository, with environment as
- OpenJDK 1.8.0_312
- Scala 2.12.15
- gramine 86f0d3f on master HEAD described by this Dockerfile
Reproduction steps go as
- Clone the project
git clone https://github.com/sammyne/gramine-quickstart-sc2.12-failure.git cd gramine-quickstart-sc2.12-failure - Dockerize the app
docker build -t gramine-quickstart-sc2.12:alpha . - Run
docker run --rm \ --device /dev/kmsg:/dev/kmsg \ --device /dev/gsgx:/dev/gsgx \ --device /dev/sgx_enclave:/dev/sgx/enclave \ --device /dev/sgx_provision:/dev/sgx/provision \ gramine-quickstart-sc2.12:alpha
Expected results
Hello, World!
Actual results
[P1:T1:] error: Error opening executable /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java: -2
[P1:T1:] error: Error during shim_init() in init_important_handles (-2)
Gramine commit hash
86f0d3f047567670dab5f8b39e4a9b9e5db97020
Looking at your manifest file (https://github.com/sammyne/gramine-quickstart-sc2.12-failure/blob/main/gramine/java.manifest.template), you seem to forgot to add some important FS mount points.
Please try to add the following ones:
[[fs.mount]]
type = "chroot"
path = "{{ entrypoint }}"
uri = "file:{{ entrypoint }}"
[[fs.mount]]
type = "chroot"
path = "/usr/local/{{ arch_libdir }}"
uri = "file:/usr/local/{{ arch_libdir }}"
[[fs.mount]]
type = "chroot"
path = "{{ java_home }}/lib/"
uri = "file:{{ java_home }}/lib/"
In general, it looks like your manifest file has incorrect FS mount points. Some of them are probably not needed at all. Some of them are duplicated. Please analyze which directories you need to expose from the host to your Java app, and tweak the FS mount points accordingly in your manifest.template file.
Looking at your manifest file (https://github.com/sammyne/gramine-quickstart-sc2.12-failure/blob/main/gramine/java.manifest.template), you seem to forgot to add some important FS mount points.
Please try to add the following ones:
[[fs.mount]] type = "chroot" path = "{{ entrypoint }}" uri = "file:{{ entrypoint }}" [[fs.mount]] type = "chroot" path = "/usr/local/{{ arch_libdir }}" uri = "file:/usr/local/{{ arch_libdir }}" [[fs.mount]] type = "chroot" path = "{{ java_home }}/lib/" uri = "file:{{ java_home }}/lib/"In general, it looks like your manifest file has incorrect FS mount points. Some of them are probably not needed at all. Some of them are duplicated. Please analyze which directories you need to expose from the host to your Java app, and tweak the FS mount points accordingly in your manifest.template file.
Adding the mount points doesn't help~ And the error remains the same =_= @dimakuv
Please show your resulting
java.manifest.sgxfile.
See https://github.com/sammyne/gramine-quickstart-sc2.12-failure/blob/main/java.manifest.sgx @dimakuv
Strange, the manifest looks fine. Are you absolutely sure that this file actually exists inside your Docker container: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java?
Can you enter into a bash session in this Docker container and check this file, like this:
ls -lh /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
file /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Please post the results of these two commands here.
ls -lh /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Gives me
-rwxr-xr-x 1 root root 15K Nov 3 13:54 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
file /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Gives me
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=aa219a43c6b3b442aea937fbb19cc8fb2ce3f346, for GNU/Linux 3.2.0, stripped
To help problem solving, I have dockerized the app as this image sammyne/gramine-quickstart-sc2.12:e9d94b2, whose working dir is /gramine. You may pull the image and check other things if you want~ @dimakuv
Ok, so the 15K size is giving me some hints...
What is the output of readelf --segments /usr/lib/jvm/java-11-openjdk-amd64/bin/java?
(Sorry, I don't want to bother with Docker images, let's continue our debugging here.)
readelf --segments /usr/lib/jvm/java-11-openjdk-amd64/bin/java
In my case of OpenJDK8, running readelf --segments /usr/lib/jvm/java-8-openjdk-amd64/bin/java produces
Elf file type is DYN (Shared object file)
Entry point 0x10b0
There are 13 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x00000000000002d8 0x00000000000002d8 R 0x8
INTERP 0x0000000000000318 0x0000000000000318 0x0000000000000318
0x000000000000001c 0x000000000000001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000718 0x0000000000000718 R 0x1000
LOAD 0x0000000000001000 0x0000000000001000 0x0000000000001000
0x0000000000000225 0x0000000000000225 R E 0x1000
LOAD 0x0000000000002000 0x0000000000002000 0x0000000000002000
0x0000000000000190 0x0000000000000190 R 0x1000
LOAD 0x0000000000002d58 0x0000000000003d58 0x0000000000003d58
0x00000000000002b8 0x00000000000002c0 RW 0x1000
DYNAMIC 0x0000000000002d68 0x0000000000003d68 0x0000000000003d68
0x0000000000000250 0x0000000000000250 RW 0x8
NOTE 0x0000000000000338 0x0000000000000338 0x0000000000000338
0x0000000000000020 0x0000000000000020 R 0x8
NOTE 0x0000000000000358 0x0000000000000358 0x0000000000000358
0x0000000000000044 0x0000000000000044 R 0x4
GNU_PROPERTY 0x0000000000000338 0x0000000000000338 0x0000000000000338
0x0000000000000020 0x0000000000000020 R 0x8
GNU_EH_FRAME 0x0000000000002040 0x0000000000002040 0x0000000000002040
0x0000000000000044 0x0000000000000044 R 0x4
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
GNU_RELRO 0x0000000000002d58 0x0000000000003d58 0x0000000000003d58
0x00000000000002a8 0x00000000000002a8 R 0x1
Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt
03 .init .plt .plt.got .plt.sec .text .fini
04 .rodata .eh_frame_hdr .eh_frame
05 .init_array .fini_array .dynamic .got .data .bss
06 .dynamic
07 .note.gnu.property
08 .note.gnu.build-id .note.ABI-tag
09 .note.gnu.property
10 .eh_frame_hdr
11
12 .init_array .fini_array .dynamic .got
@dimakuv
@pwmarcz Maybe you have any ideas? I don't see anything wrong with this file and the manifest.
I reproduced the steps, and from the logs it's apparent that Gramine does not mount anything from manifest:
[::] debug: mounting "file:." (chroot) under /
[::] debug: mounting "proc" (pseudo) under /proc
[::] debug: mounting "dev" (pseudo) under /dev
[::] debug: mounting "dev:tty" (chroot) under /dev/tty
This is because the manifest uses fs.mount, but the new syntax is fs.mounts.
I reproduced the steps, and from the logs it's apparent that Gramine does not mount anything from manifest:
[::] debug: mounting "file:." (chroot) under / [::] debug: mounting "proc" (pseudo) under /proc [::] debug: mounting "dev" (pseudo) under /dev [::] debug: mounting "dev:tty" (chroot) under /dev/ttyThis is because the manifest uses
fs.mount, but the new syntax isfs.mounts.
Changing fs.mount as fs.mounts works! @pwmarcz Great thanks 👍
Recently, without any change on codes, the app would be stuck with logs as
Gramine is starting. Parsing TOML manifest file, this may take some time...
debug: Gramine parsed TOML manifest file successfully
debug: Token file: java.token
debug: Read dummy DCAP token
debug: Enclave created:
debug: base: 0x0000000000000000
debug: size: 0x0000000400000000
debug: misc_select: 0x00000000
debug: attr.flags: 0x0000000000000005
debug: attr.xfrm: 0x0000000000000007
debug: ssa_frame_size: 4
debug: isv_prod_id: 0x00000000
debug: isv_svn: 0x00000000
debug: Adding pages to SGX enclave, this may take some time...
debug: Adding pages to enclave: 0x3fff47000-0x400000000 [REG:R--] (manifest) measured
debug: Adding pages to enclave: 0x3ffd47000-0x3fff47000 [REG:RW-] (ssa) measured
debug: Adding pages to enclave: 0x3ffd07000-0x3ffd47000 [TCS:---] (tcs) measured
debug: Adding pages to enclave: 0x3ffcc7000-0x3ffd07000 [REG:RW-] (tls) measured
debug: Adding pages to enclave: 0x3ffc87000-0x3ffcc7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffc47000-0x3ffc87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffc07000-0x3ffc47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffbc7000-0x3ffc07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffb87000-0x3ffbc7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffb47000-0x3ffb87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffb07000-0x3ffb47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffac7000-0x3ffb07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffa87000-0x3ffac7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffa47000-0x3ffa87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ffa07000-0x3ffa47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff9c7000-0x3ffa07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff987000-0x3ff9c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff947000-0x3ff987000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff907000-0x3ff947000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff8c7000-0x3ff907000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff887000-0x3ff8c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff847000-0x3ff887000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff807000-0x3ff847000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff7c7000-0x3ff807000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff787000-0x3ff7c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff747000-0x3ff787000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff707000-0x3ff747000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff6c7000-0x3ff707000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff687000-0x3ff6c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff647000-0x3ff687000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff607000-0x3ff647000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff5c7000-0x3ff607000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff587000-0x3ff5c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff547000-0x3ff587000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff507000-0x3ff547000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff4c7000-0x3ff507000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff487000-0x3ff4c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff447000-0x3ff487000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff407000-0x3ff447000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff3c7000-0x3ff407000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff387000-0x3ff3c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff347000-0x3ff387000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff307000-0x3ff347000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff2c7000-0x3ff307000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff287000-0x3ff2c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff247000-0x3ff287000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff207000-0x3ff247000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff1c7000-0x3ff207000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff187000-0x3ff1c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff147000-0x3ff187000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff107000-0x3ff147000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff0c7000-0x3ff107000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff087000-0x3ff0c7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff047000-0x3ff087000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3ff007000-0x3ff047000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fefc7000-0x3ff007000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fef87000-0x3fefc7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fef47000-0x3fef87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fef07000-0x3fef47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3feec7000-0x3fef07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fee87000-0x3feec7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fee47000-0x3fee87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fee07000-0x3fee47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fedc7000-0x3fee07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fed87000-0x3fedc7000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fed47000-0x3fed87000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fed07000-0x3fed47000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fecc7000-0x3fed07000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0x3fecb7000-0x3fecc7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feca7000-0x3fecb7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec97000-0x3feca7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec87000-0x3fec97000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec77000-0x3fec87000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec67000-0x3fec77000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec57000-0x3fec67000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec47000-0x3fec57000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec37000-0x3fec47000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec27000-0x3fec37000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec17000-0x3fec27000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fec07000-0x3fec17000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3febf7000-0x3fec07000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3febe7000-0x3febf7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3febd7000-0x3febe7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3febc7000-0x3febd7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3febb7000-0x3febc7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feba7000-0x3febb7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb97000-0x3feba7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb87000-0x3feb97000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb77000-0x3feb87000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb67000-0x3feb77000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb57000-0x3feb67000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb47000-0x3feb57000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb37000-0x3feb47000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb27000-0x3feb37000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb17000-0x3feb27000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feb07000-0x3feb17000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feaf7000-0x3feb07000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feae7000-0x3feaf7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fead7000-0x3feae7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feac7000-0x3fead7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feab7000-0x3feac7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3feaa7000-0x3feab7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea97000-0x3feaa7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea87000-0x3fea97000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea77000-0x3fea87000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea67000-0x3fea77000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea57000-0x3fea67000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea47000-0x3fea57000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea37000-0x3fea47000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea27000-0x3fea37000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea17000-0x3fea27000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fea07000-0x3fea17000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9f7000-0x3fea07000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9e7000-0x3fe9f7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9d7000-0x3fe9e7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9c7000-0x3fe9d7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9b7000-0x3fe9c7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe9a7000-0x3fe9b7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe997000-0x3fe9a7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe987000-0x3fe997000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe977000-0x3fe987000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe967000-0x3fe977000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe957000-0x3fe967000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe947000-0x3fe957000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe937000-0x3fe947000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe927000-0x3fe937000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe917000-0x3fe927000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe907000-0x3fe917000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe8f7000-0x3fe907000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe8e7000-0x3fe8f7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe8d7000-0x3fe8e7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe8c7000-0x3fe8d7000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0x3fe4a2000-0x3fe4ef000 [REG:R-X] (code) measured
debug: Adding pages to enclave: 0x3fe4ef000-0x3fe4f3000 [REG:RW-] (data) measured
debug: Adding pages to enclave: 0x3fe4f3000-0x3fe8c7000 [REG:RW-] (bss) measured
debug: Adding pages to enclave: 0x10000-0x3fe4a2000 [REG:RWX] (free)
debug: Added all pages to SGX enclave
debug: Enclave initializing:
debug: enclave id: 0x00000003fffff000
debug: mr_enclave: 39dd3804da67ab3c70e6ab1a0bad9abac93174c2baad7d05c05307f93e21322a
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:
- loader.log_level = warning|debug|trace|all (verbose log level, may leak information)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------
warning: DkVirtualMemoryProtect is unimplemented in Linux-SGX PAL
[::] debug: Gramine was built from commit: 86f0d3f047567670dab5f8b39e4a9b9e5db97020
[::] debug: Host: Linux-SGX
[::] debug: LibOS xsave_enabled 1, xsave_size 0x340(832), xsave_features 0x7
[::] debug: Initial VMA region 0x3fde00000-0x3fde9e000 (LibOS) bookkeeped
[::] debug: Initial VMA region 0x3fff47000-0x400000000 (manifest) bookkeeped
[::] debug: ASLR top address adjusted to 0x2db9a9000
[::] debug: host is Linux-SGX, adding SGX-specific /dev/attestation files: report, quote, etc.
[::] debug: Shim loaded at 0x3fde00000, ready to initialize
[::] debug: mounting "file:." (chroot) under /
[::] debug: mounting "proc" (pseudo) under /proc
[::] debug: mounting "dev" (pseudo) under /dev
[::] debug: mounting "dev:tty" (chroot) under /dev/tty
[::] debug: Creating pipe: pipe.srv:2b4363aec509f8b50d2f931a230f1d5c7d1016cac5a814ff14f0b83a537d4aac
[P1:T1:] debug: mounting "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc" (chroot) under /lib
[P1:T1:] debug: mounting "file:/lib/x86_64-linux-gnu" (chroot) under /lib/x86_64-linux-gnu
[P1:T1:] debug: mounting "file:/usr//lib/x86_64-linux-gnu" (chroot) under /usr//lib/x86_64-linux-gnu
[P1:T1:] debug: mounting "file:/bin" (chroot) under /bin
[P1:T1:] debug: mounting "file:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" (chroot) under /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
[P1:T1:] debug: mounting "file:/usr/local//lib/x86_64-linux-gnu" (chroot) under /usr/local//lib/x86_64-linux-gnu
[P1:T1:] debug: mounting "file:/usr/lib/jvm/java-8-openjdk-amd64" (chroot) under /usr/lib/jvm/java-8-openjdk-amd64
[P1:T1:java] debug: Creating pipe: pipe.srv:9b0acca020707434150cda568ca0b521a2b2b5714bbe1c9d077a3ee48a3c43af
[P1:T1:java] debug: Allocating stack at 0 (size = 262144)
[P1:T1:java] debug: loading "file:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java"
[P1:T1:java] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2
[P1:T1:java] debug: loading "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:java] debug: Creating pipe: pipe.srv:1
[P1:T1:java] debug: Creating pipe: pipe.srv:7b88e56153f05da760e621b6e6d7446fe98702a83da416f5699a5322432a8593
[P1:T1:java] debug: Shim process initialized
[P1:shim] debug: IPC worker started
[P1:T1:java] warning: Not supported flag (0x3001) passed to arch_prctl
[P1:T1:java] debug: glibc register library /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/../lib/amd64/jli/libjli.so loaded at 0x2db91a000
[P1:T1:java] debug: glibc register library /lib/libc.so.6 loaded at 0x2db720000
[P1:T1:java] debug: glibc register library /lib/x86_64-linux-gnu/libz.so.1 loaded at 0x2db704000
[P1:T1:java] debug: glibc register library /lib/libdl.so.2 loaded at 0x2db6ff000
[P1:T1:java] debug: glibc register library /lib/libpthread.so.0 loaded at 0x2db6fa000
[P1:T1:java] debug: glibc register library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so loaded at 0x2da8f6000
[P1:T1:java] debug: glibc register library /lib/x86_64-linux-gnu/libstdc++.so.6 loaded at 0x2da714000
[P1:T1:java] debug: glibc register library /lib/libm.so.6 loaded at 0x2da639000
[P1:T1:java] debug: glibc register library /lib/x86_64-linux-gnu/libgcc_s.so.1 loaded at 0x2da61e000
[P1:T1:java] warning: Unsupported system call 435
[P1:T1:java] debug: Creating pipe: pipe.srv:6975cd48d48661cc7f29e72fc72ac2cc1ce01ec2a6e7bdc7b3767d0a4f1af1a7
[P1:T1:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T1:java] warning: Non-private futexes are not supported, assuming implicit FUTEX_PRIVATE_FLAG
[P1:T2:java] debug: ipc send to 1: IPC_MSG_PID_GETMETA(2, CRED)
[P1:T2:java] debug: Sending ipc message to 1
[P1:T2:java] debug: Waiting for a response to 1
[P1:shim] debug: IPC worker: received IPC message from 1: code=8 size=25 seq=1
[P1:shim] debug: ipc callback from 1: IPC_MSG_PID_GETMETA(2, CRED)
[P1:shim] debug: IPC send to 1: shim_ipc_pid_retmeta{8, ...}
[P1:shim] debug: Sending ipc message to 1
[P1:shim] debug: IPC worker: received IPC message from 1: code=0 size=37 seq=1
[P1:shim] debug: Got an IPC response from 1, seq: 1
[P1:T2:java] debug: Waiting finished: 0
[P1:T2:java] debug: glibc register library /lib/librt.so.1 loaded at 0x2da518000
[P1:T2:java] debug: glibc register library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so loaded at 0x2da505000
[P1:T2:java] debug: glibc register library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so loaded at 0x2da4d6000
[P1:T2:java] debug: glibc register library /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so loaded at 0x2da4c0000
[P1:T2:java] debug: memory fault at 0x00000000 (IP = 0x2c50002b4)
[P1:T2:java] debug: Created sigframe for sig: 11 at 0x2da61c150 (handler: 0x2db271ff0, restorer: 0x2db761b40)
[P1:T2:java] warning: Unsupported system call 435
[P1:T2:java] debug: Creating pipe: pipe.srv:1f71853e03a7a4b21adfa189411dd1637950cfae63b6485ca79aca4c5786327b
[P1:T2:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T3:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T2:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T3:java] warning: Unsupported system call prctl
[P1:T2:java] warning: Unsupported system call 435
[P1:T2:java] debug: Creating pipe: pipe.srv:dedb093a588bedc2f1c10d0c138dd67a04d3ed0e46da8e777f0cbb4340084bb2
[P1:T2:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T4:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T2:java] warning: Unsupported system call 435
[P1:T2:java] debug: Creating pipe: pipe.srv:43f0b076c47067f2c042f338864f04eed5ec45aca2d99614509150bd4df483d0
[P1:T2:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T5:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T4:java] warning: Unsupported system call prctl
[P1:T4:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T5:java] warning: Unsupported system call prctl
[P1:T5:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T5:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T2:java] warning: Unsupported system call 435
[P1:T2:java] debug: Creating pipe: pipe.srv:e6614604b66a8e53bf1cab3caa2491bac89a028b912c7631d1dcbebf968f0478
[P1:T2:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T6:java] warning: Ignoring FUTEX_CLOCK_REALTIME flag
Helps, please~ @pwmarcz @dimakuv
@sammyne There must have been such change... Did anything change?
Also, does this hang happen every time, or only sporadically?
@sammyne There must have been such change... Did anything change?
Also, does this hang happen every time, or only sporadically?
I'm pretty certain that nothing changes, since my app is always packed and run as a docker image. Besides, the hang happened before, and I tackle that with restarting the container. But recently, using exactly the same docker image in the same machine, the hang happens very frequently, So the issue is re-opened for asking helps= =
Occasional successful run produces sample log as ok.log. Frequent failed run produces partial sample log (with tailing lines repeat over and over again) as err.log.
@dimakuv
I'm debugging same issue during these days.
In my case, I run org.apache.spark.examples.SparkPi from spark-3.1.2-bin-hadoop2.7. It fails randomly every time at very early stage.
I tried both jdk1.8.0_192 and java-8-openjdk-amd64. gramine-sgx and gramine-direct both reproduce this issue. Attached a log of gramine-direct
spark-pi.log
grep "\.class" spark-pi.log | wc -l can return how many classes are tried to open. It can be from 10 to 1200 in different runs.
Every time, Thread 2 of the single process Java app hangs on a pthread_cond_wait
[P1:T2:java] trace: ---- shim_futex(0x30fab4013d7c, FUTEX_PRIVATE|FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET, 0, 0, 0, -1) ...
when loaded classes/jars.
I wrote some test cases to test pthread_cond_* and clock_gettime, but haven't reproduce this issue using C code.
I would start with investigating why [P1:T2:java] debug: memory fault at 0x00000000 (IP = 0x2c50002b4) happens
@llly
Hi, I am recently trying to run Spark applications in Gramine-SGX, but it seems to need non-trivial efforts. Did you manage to run the Spark? Could you please share the write-up for running Spark?
I'm closing this issue since it is inactive for a year. @sammyne If you still need help, feel free to reopen the issue (and provide new details).