write exists encrypted file error

[mccoysc]

Step to re-produce

1，make empty directory to be encrypted mounts with key_name="_sgx_mrenclave" 2,make and run the result app. 3,the result app do the following: a,create a file in the encrypted directory and write "hello" to the file,close and exit the app. b,re-do the above,the app crashed.

Expected results

file created and encrypted and the "re-do" also work well.

Actual results

the "re-do" cause app crashed.

debug: load_protected_file: encrypted/hello.txt, fd 43, size 4096, mode 1, create 0, pf 0x7e1594b0 debug: load_protected_file: encrypted/hello.txt, fd 43: opening new PF 0x7e1594b0 debug: ipf_open: handle: 43, path: 'encrypted/hello.txt', real size: 4096, mode: 0x1 debug: ipf_init_existing_file: data size 5 debug: ipf_open: OK (data size 5) debug: ipf_internal_flush: no need to write error: Unexpected memory fault occurred inside PAL (pf_get_size at protected_files.c, libpal.so+0x39bc0) debug: (untrusted PAL sent PAL event 0x2) debug: DkProcessExit: Returning exit code 1

[mccoysc]

this is the templete to make app:

[loader] entrypoint = "file:{{ gramine.libos }}" log_level = "{{ log_level }}" insecure__use_cmdline_argv = true insecure__use_host_env = true

[loader.env] LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"

[libos] entrypoint="{{ nodejs_dir }}/node"

[sys] insecure__allow_eventfd = true

[fs] mounts=[ {type = "chroot",path = "{{ nodejs_dir }}/node",uri = "file:{{ nodejs_dir }}/node"}, {type = "chroot",path = "/lib",uri = "file:{{ gramine.runtimedir()}}"}, {type = "chroot",path = "{{ arch_libdir }}",uri = "file:{{ arch_libdir }}"}, {type = "chroot",path = "/usr/{{ arch_libdir }}",uri = "file:/usr/{{ arch_libdir }}"}, {type = "encrypted",path = "/encrypted",uri = "file:encrypted",key_name="_sgx_mrenclave"} ]

[sgx] nonpie_binary = true enclave_size = "2G" thread_num = 64 remote_attestation = false debug = true

trusted_files=[ "file:{{ gramine.libos }}", "file:{{ nodejs_dir }}/", "file:{{ gramine.runtimedir()}}/", "file:{{ arch_libdir }}/", "file:/usr/{{ arch_libdir }}/" ]

allowed_files=[ "file:helloworld.js" ]

protected_mrenclave_files = [ "file:encrypted", ]

[mccoysc]

the first run is ok but the later would crash always. if i delete the encrypted file "hello.txt",it will work well but repeat the crash after that. this is the helloword.js:

console.log("Hello World"); var fs=require("fs") var data=fs.readFileSync("/dev/attestation/keys/_sgx_mrenclave"); console.log("mrenclave key:",data.toString("hex")) fs.writeFileSync("/encrypted/hello.txt","hello"); console.log(fs.readFileSync("/encrypted/hello.txt").toString())。

[mccoysc]

@stefanberger

[lejunzhu]

I tried to run the helloworld code on nodejs 14 and latest Gramine. It doesn't crash, and the log is:

[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] warning: encrypted_file_set_size: pf_set_size failed: Functionality not implemented
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0666) = -13

The error happens when the application tries to truncate the encrypted file. This is not implemented now.

https://github.com/gramineproject/gramine/blob/01e01513e08b0d8ec70f074dda6de8f539e4611f/common/src/protected_files/protected_files.c#L1254

As a workaround, if you can modify the application code, you can try to delete the files before opening them with O_TRUNC.

[mccoysc]

I tried to run the helloworld code on nodejs 14 and latest Gramine. It doesn't crash, and the log is:

[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] warning: encrypted_file_set_size: pf_set_size failed: Functionality not implemented
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0666) = -13

The error happens when the application tries to truncate the encrypted file. This is not implemented now.

https://github.com/gramineproject/gramine/blob/01e01513e08b0d8ec70f074dda6de8f539e4611f/common/src/protected_files/protected_files.c#L1254

As a workaround, if you can modify the application code, you can try to delete the files before opening them with O_TRUNC.

yes,i got the answer after reading the code. but another bug: if i use the code in node.js as follow: fs.appendFileSync to append data to the end of an encrypted file,data is written to the head of the file(expect wirtten to the end of file)

[mccoysc]

really,fs.appendFileSync equals to fopen('file','a');fwrite(,some_data); so data append to the file is expected.

[dimakuv]

@lejunzhu Have you encountered/debugged this issue with fs.appendFileSync()? Looks like a genuine bug...

[lejunzhu]

@lejunzhu Have you encountered/debugged this issue with fs.appendFileSync()? Looks like a genuine bug...

According to this comment, Gramine doesn't support O_APPEND today:

https://github.com/gramineproject/gramine/blob/d6dda2191184123cbc1897267e0404dbe4b77e90/LibOS/shim/include/shim_flags_conv.h#L35

I think I saw this problem with "allowed files" before. Probably not an encrypted FS specific issue.

[mccoysc]

i have written a c code like this: FILE* file=fopen("encrypted_file","a");//encrypted_file exists already and have content "hello". fwrite("world",1,5,file); fclose(file); the file content is expected to be "helloworld". but,the content is only "world",the "hello" is overwritten.

[mccoysc]

@lejunzhu Have you encountered/debugged this issue with fs.appendFileSync()? Looks like a genuine bug...

another another bug: fs.existsSync always get returned value "false". this function is used to be sure "the file/path exists?"

[lejunzhu]

another another bug: fs.existsSync always get returned value "false". this function is used to be sure "the file/path exists?"

Here's what I got with commit 01e01513e08b0d8ec70f074dda6de8f539e4611f. Could you please try again after updating Gramine?

Test code:

var fs=require("fs")
console.log(fs.existsSync("mrenclave/hello.txt"));
fs.writeFileSync("mrenclave/hello.txt","hello");
console.log(fs.existsSync("mrenclave/hello.txt"));
console.log(fs.readFileSync("mrenclave/hello.txt").toString())

Log:

...
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = -2
...
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9e0, 0x6) ...
false
...
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 0, mode: 0x3
[P1:T1:node] debug: ipf_open: OK (data size 0)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0666) = 0x11
[P1:T1:node] trace: ---- shim_write(17, 0x1f60e9e8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9f0, 0x5) ...
true
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_RDONLY|0x80000, 0000) = 0x11
[P1:T1:node] trace: ---- shim_fstat(17, 0x1fda50c0) = 0x0
[P1:T1:node] trace: ---- shim_read(17, 0x1f60e9f8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x5
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60ea00, 0x6) ...
hello

The result seems correct: the first test returns "false" and the second returns "true".

[mccoysc]

ok,i'll test again

[dimakuv]

According to this comment, Gramine doesn't support O_APPEND today:

@lejunzhu No, that's not like this. Gramine's PAL component doesn't support O_APPEND. But Gramine's LibOS component supports O_APPEND (at least from what I remember). So, LibOS emulates O_APPEND on top of PAL, without the PAL actually knowing that the file is opened in "append" mode (this happens by the LibOS layer updating the offset value and sending this offset value ot PAL, so that PAL knows at which offset to start writing in the file).

At least that's how it is supposed to work. But it looks like we have some bug in the Gramine LibOS implementation -- at least for Protected/Encrypted Files.

[mccoysc]

another another bug: fs.existsSync always get returned value "false". this function is used to be sure "the file/path exists?"

Here's what I got with commit 01e0151. Could you please try again after updating Gramine?

Test code:

var fs=require("fs")
console.log(fs.existsSync("mrenclave/hello.txt"));
fs.writeFileSync("mrenclave/hello.txt","hello");
console.log(fs.existsSync("mrenclave/hello.txt"));
console.log(fs.readFileSync("mrenclave/hello.txt").toString())

Log:

...
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = -2
...
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9e0, 0x6) ...
false
...
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 0, mode: 0x3
[P1:T1:node] debug: ipf_open: OK (data size 0)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0666) = 0x11
[P1:T1:node] trace: ---- shim_write(17, 0x1f60e9e8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9f0, 0x5) ...
true
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_RDONLY|0x80000, 0000) = 0x11
[P1:T1:node] trace: ---- shim_fstat(17, 0x1fda50c0) = 0x0
[P1:T1:node] trace: ---- shim_read(17, 0x1f60e9f8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x5
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60ea00, 0x6) ...
hello

The result seems correct: the first test returns "false" and the second returns "true".

ok,i test again and it works well

[lejunzhu]

At least that's how it is supposed to work. But it looks like we have some bug in the Gramine LibOS implementation -- at least for Protected/Encrypted Files.

I tested it again, and in latest Gramine, allowed files have the same behavior (ignoring O_APPEND) as well.

[mccoysc]

another another bug: fs.existsSync always get returned value "false". this function is used to be sure "the file/path exists?"

Here's what I got with commit 01e0151. Could you please try again after updating Gramine?

Test code:

var fs=require("fs")
console.log(fs.existsSync("mrenclave/hello.txt"));
fs.writeFileSync("mrenclave/hello.txt","hello");
console.log(fs.existsSync("mrenclave/hello.txt"));
console.log(fs.readFileSync("mrenclave/hello.txt").toString())

Log:

...
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = -2
...
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9e0, 0x6) ...
false
...
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 0, mode: 0x3
[P1:T1:node] debug: ipf_open: OK (data size 0)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0666) = 0x11
[P1:T1:node] trace: ---- shim_write(17, 0x1f60e9e8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_access("mrenclave/hello.txt", F_OK) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60e9f0, 0x5) ...
true
[P1:T1:node] trace: ---- return from shim_write(...) = 0x5
[P1:T1:node] debug: ipf_open: handle: 0, path: 'mrenclave//hello.txt', real size: 4096, mode: 0x3
[P1:T1:node] debug: ipf_init_existing_file: data size 5
[P1:T1:node] debug: ipf_open: OK (data size 5)
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "mrenclave/hello.txt", O_RDONLY|0x80000, 0000) = 0x11
[P1:T1:node] trace: ---- shim_fstat(17, 0x1fda50c0) = 0x0
[P1:T1:node] trace: ---- shim_read(17, 0x1f60e9f8, 0x5) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x5
[P1:T1:node] debug: ipf_internal_flush: no need to write
[P1:T1:node] trace: ---- shim_close(17) = 0x0
[P1:T1:node] trace: ---- shim_write(1, 0x1f60ea00, 0x6) ...
hello

The result seems correct: the first test returns "false" and the second returns "true".

3rd another : if an "_sgx_mrenclave" encrypted file named enc_file is created by enclave "a", the code "fs.existsSync('enc_file')" in enclave "b" would always return false and fs.writeFileSync("enc_file") in enclave "b" always get crashed.

[llly]

if an "_sgx_mrenclave" encrypted file named enc_file is created by enclave "a", the code "fs.existsSync('enc_file')" in enclave "b" would always return false and fs.writeFileSync("enc_file") in enclave "b" always get crashed.

I did a quick test. Behavior of encrypted file is expected. When mrenclave doesn't match, EACCES: permission denied is returned by open. nodejs need to handle this error. To reproduce it, run chown and chmod to remove read permission. Then run a bare nodejs helloworld.js and will see same crash/exception

 errno: -13,
  syscall: 'open',
  code: 'EACCES',

[mkow]

Not a bug in Gramine (see @llly's comment above), closing.