flush works incorrectly

[g302ge]

Description of the problem

in my software, I am try use the File in gramine SGX env, the File could not flush into file. But the same binary in gramine direct mode could flush normally

Steps to reproduce

the software work logic is that

let file = open("path/to/file");
file.write("helloworld");
file.flush();

let buffer = file.read_all();
println(buffer);

Expected results

flush normally

Actual results

In direct mode:

But in SGX mode

Gramine commit hash

7846eccb5c226e9c6aa8775058d43506c6ad7cfd

[mkow]

dimakuv/noop-fsync-on-dir

I don't know this branch. Please verify the issues before reporting on the current master, as the issue template asks to ;)

[g302ge]

Sure, but I belive that this bug will raise agin at the master branch(cc I have tried at master branch )

[meithecatte]

Could you provide a complete minimal reproducer? The code you've put into the "Steps to reproduce" is not a full program. I'm not even sure what language your example is in.

[g302ge]

Could you provide a complete minimal reproducer? The code you've put into the "Steps to reproduce" is not a full program. I'm not even sure what language your example is in.

I wrote my software using Rust, and the write into file flow is below

1. open file waiting for writing
2. write some data with the handle
3. flush file handle

But after flusing, the file only contains the gramine procted file metadata, without any data I have wrote before.

Unluckly, the real code is too big to put on this issue, I think there is a bug around the file flushing logic in gramine. I am confused why the flus operation no use in gramine env

[meithecatte]

Surely the full code of your application is not required to reproduce the issue. It shouldn't be too difficult to remove most stuff and get a reasonably-sized example that demonstrates the problem.

[g302ge]

Well, the software is LevelDB rust version. I try to split the core IO logic from the LevelDB

[meithecatte]

Based on your "steps to reproduce", I have created the following files:

repro.rs:

use std::error::Error;
use std::fs::File;
use std::io::prelude::*;

fn main() -> Result<(), Box<dyn Error>> {
    {
        let mut file = File::create("/tmp/testing")?;
        file.write(b"Hello world")?;
        file.flush()?;
    }

    let data = std::fs::read_to_string("/tmp/testing")?;
    println!("{:?}", data);
    Ok(())
}

repro.manifest.template:

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "repro"
loader.log_level = "{{ log_level }}"
loader.argv0_override = "repro"

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}"

fs.mounts = [
  { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
  { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
  { path = "/tmp", type = "tmpfs" },
]

sgx.debug = true
sgx.nonpie_binary = true

sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:repro",
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/"
]

Makefile:

ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)
CFLAGS = -Wall -Wextra

ifeq ($(DEBUG),1)
GRAMINE_LOG_LEVEL = debug
CFLAGS += -g
else
GRAMINE_LOG_LEVEL = error
CFLAGS += -O3
endif

.PHONY: all
all: repro repro.manifest
ifeq ($(SGX),1)
all: repro.manifest.sgx repro.sig repro.token
endif

repro: repro.rs
	rustc repro.rs

repro.manifest: repro.manifest.template
	gramine-manifest \
		-Dlog_level=$(GRAMINE_LOG_LEVEL) \
		-Darch_libdir=$(ARCH_LIBDIR) \
		$< $@

repro.manifest.sgx: repro.manifest repro
	gramine-sgx-sign \
		--manifest $< \
		--output $@

repro.sig: repro.manifest.sgx

repro.token: repro.sig
	gramine-sgx-get-token \
		--output $@ --sig $<

ifeq ($(SGX),)
GRAMINE = gramine-direct
else
GRAMINE = gramine-sgx
endif

.PHONY: clean
clean:
	$(RM) *.token *.sig *.manifest.sgx *.manifest repro

.PHONY: distclean
distclean: clean

I then ran make SGX=1 and gramine-sgx repro. This did not reproduce the bug — the program printed "Hello world" as expected. Please provide more accurate reproduction instructions.

[g302ge]

@NieDzejkob I suggest use my code in https://github.com/g302ge/leveldb-rs/tree/master/examples/gramine

Steps:

1. make SGX=1
2. gramine-sgx gramine

in my code this will produce a empty result as I mentationed.

But I directly use cargo run anything will be OK

[g302ge]

But I use the protected_mrenclave_file

[dimakuv]

@g302ge I looked at our Protected Files code, and I don't see any bug in there. I'm unsure why your code wouldn't flush the content of the file.

In your original post, you showed a screenshot of the "read the file again" part of Gramine-SGX log. Could you attach the complete Gramine-SGX log here, so we can also check what happens during the "flush the file" part?

[g302ge]

emmm, I think the protected_file flush could not worl well at current master version

[libos]
entrypoint = "gramine"

[loader]
preload = "file:{{ gramine.libos }}"
entrypoint = "file:{{ gramine.libos }}"
log_level = "{{ log_level }}"
argv0_override = "gramine"

[loader.env]
LD_LIBRARY_PATH = "/lib:/lib/x86_64-linux-gnu"
RUST_BACKTRACE = "full"

[fs]
mounts = [
    { type = "chroot", path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
    { type = "chroot", path = "/lib/x86_64-linux-gnu", uri = "file:/lib/x86_64-linux-gnu" },
    { type = "chroot", path = "/data", uri = "file:data" },
    { type = "chroot", path = "/allowed", uri = "file:allowed" },
]


[sgx]
debug = true
enclave_size = "2G"
thread_num = 32
trusted_files = [
    "file:{{ gramine.libos }}",
    "file:{{ gramine.runtimedir() }}/",
    "file:gramine",
    "file:/lib/x86_64-linux-gnu/",
]
allowed_files = [
    "file:allowed/"
]
protected_mrenclave_files = [
    "file:data/"
]

I want flush a file in data dir, but in my any test it does not work well for example :

I have mounted the data directory at the point /data and then I have wrote some contents into the /data/MANIFEST-000001 file, Though I have call the flush method , I can't get any content I have wrote

[g302ge]

And more , the log I got from gramine runtime is below , just flush part

[P1:T1:gramine] trace: ---- shim_openat(AT_FDCWD, "/data/simple_db_workds/MANIFEST-000001", O_WRONLY|O_CREAT|0x80000, 0666) = 0x3
 [P1:T1:gramine] trace: ---- shim_write(1, 0xcb2ee180, 0xa7) ...
 LOGGER RECORD is [1, 26, 108, 101, 118, 101, 108, 100, 98, 46, 66, 121, 116, 101, 119, 105, 115, 101, 67, 111, 109, 112, 97, 114, 97, 116, 111, 114, 2,       0, 3, 2, 4, 0]
[P1:T1:gramine] trace: ---- return from shim_write(...) = 0xa7
P1:T1:gramine] trace: ---- shim_write(3, 0xcb2edff0, 0x4) ...
[P1:T1:gramine] trace: ---- return from shim_write(...) = 0x4
 [P1:T1:gramine] trace: ---- shim_write(3, 0xc96489a0, 0x2) ...
[P1:T1:gramine] trace: ---- return from shim_write(...) = 0x2
P1:T1:gramine] trace: ---- shim_write(3, 0xc96489f0, 0x1) ...
[P1:T1:gramine] trace: ---- return from shim_write(...) = 0x1                                                                                           
[P1:T1:gramine] trace: ---- shim_write(3, 0xcb2ee070, 0x22) ...
 [P1:T1:gramine] trace: ---- return from shim_write(...) = 0x22
 [P1:T1:gramine] trace: ---- shim_write(1, 0xcb2ee180, 0x17) ...
 LOGGER FLUSH is Ok(())

Result is fine ,but real thing is that NOTHING flush into the file

[g302ge]

@g302ge I looked at our Protected Files code, and I don't see any bug in there. I'm unsure why your code wouldn't flush the content of the file.

In your original post, you showed a screenshot of the "read the file again" part of Gramine-SGX log. Could you attach the complete Gramine-SGX log here, so we can also check what happens during the "flush the file" part?

I think you could try the case flushing data into a procted_mrenclave_files

[dimakuv]

I'm closing this issue as it was inactive for almost a year. @g302ge If you still experience this issue, please reopen the issue and provide us with additional details.