gramine
gramine copied to clipboard
gramineproject
warning: encrypted_file_internal_open: pf_open failed: Invalid path
Description of the problem
I am trying to run a ML model where the inputs to the model are encrypted (kind of like the pytorch example in the docs, but more involved). I have run the protected files client that comes with gramine with no issues, and also the pytorch example in the docs.
For my current model, I encrypted all of the input image files using the wrap key, and mount the input directory as an encrypted directory.
The DCAP remote attestation works ok and my enclave is able to receive the wrap key, however, when the model tries to open the actual files, I get the following error:
warning: encrypted_file_internal_open: pf_open failed: Invalid path
Here are the some more log messages:
[P1:T1:python3] trace: ---- shim_stat("./datasets/cd-sem-enc/AB/train", 0x1c6980bd
0) = 0x0
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "./datasets/cd-sem-enc/AB/train"
, O_RDONLY|0x90800, 0000) = 0x3
[P1:T1:python3] trace: ---- shim_newfstatat(3, "", 0x1c6980af0, 4096) = 0x0
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
the ./datasets/cd-sem-enc/AB/train directory is the directory holding the encrypted images. I've mounted it like so:
# need this line for compatibility reasons
# loader.preload = "file:{{ gramine.libos }}"
loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"
loader.insecure__use_cmdline_argv = true
loader.log_level = "{{ log_level }}"
loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/lib:/usr/bin/:/usr/{{ arch_libdir }}"
loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "localhost:4433"
loader.argv0_override = "{{ entrypoint }}"
fs.mounts = [
{ path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
{ path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
{ path = "/datasets/cd-sem-enc/", uri = "file:datasets/cd-sem-enc/", type = "encrypted"},
{ path = "/etc", uri = "file:/etc" },
{ path = "/usr", uri = "file:/usr" },
{ path = "/tmp", uri = "file:/tmp" },
{ path = "/usr/lib", uri = "file:/usr/lib" },
]
loader.pal_internal_mem_size = "256M"
sgx.debug = true
sgx.thread_num = 10
sgx.nonpie_binary = true
sgx.enclave_size = "8192M"
sgx.remote_attestation = "dcap"
# make sure you install numpy, torch and torchvision as sudo so they are installed to the distlib not the local user
sgx.trusted_files = [
"file:{{ gramine.libos }}",
"file:{{ gramine.runtimedir() }}/",
"file:{{ entrypoint }}",
"file:{{ arch_libdir }}/",
"file:/usr{{ arch_libdir }}/",
"file:{{ python.stdlib }}/",
"file:{{ python.distlib }}/",
"file:/usr/local/lib/python3.8/dist-packages/",
"file:/usr/lib/python3/dist-packages/",
"file:{{python.get_path('stdlib', vars={'installed_base': '/usr/local'}) }}/",
"file:ssl/ca.crt",
"file:/usr/lib/python3.8/",
"file:{{ env.HOME }}/gramine/CI-Examples/pix2pix/",
"file:./"
]
sgx.allowed_files = [
"file:/etc/nsswitch.conf",
"file:/etc/ethers",
"file:/etc/host.conf",
"file:/etc/hosts",
"file:/etc/group",
"file:/etc/passwd",
"file:/etc/gai.conf",
"file:/etc/resolv.conf",
"file:/usr/share/dpkg/cputable",
"file:/usr/share/dpkg/tupletable",
"file:/etc/apt/apt.conf.d/",
"file:/usr/lib/ssl/openssl.cnf",
"file:/etc/default/apport",
"file:/tmp/", #to cache stuff
"file:./checkpoints/", #cd_sem_pix2pix_enc/"
"file:./results/",
]
Steps to reproduce
run make train-cd with the following Makefile
#!/bin/bash
GRAMINE_LOG_LEVEL ?= all
ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)
ENTRYPOINT ?= python3
SGX_SIGNER_KEY ?= /home/trishita/.config/gramine/enclave-key.pem
CFLAGS = -Wall -Wextra -O3
CC = gcc
SGX = 1
# note that entry point needs to be full path
python3.manifest: python3.manifest.template train.py #~/gramine/CI-Examples/io-secure/input.txt
gramine-manifest \
-Dentrypoint=/usr/bin/$(ENTRYPOINT) \
-Darch_libdir=$(ARCH_LIBDIR) \
-Dlog_level=$(GRAMINE_LOG_LEVEL) \
$< $@
python3.manifest.sgx: python3.manifest
@test -s $(SGX_SIGNER_KEY) || \
{ echo "SGX signer private key was not found, please specify SGX_SIGNER_KEY!"; exit 1; }
sudo gramine-sgx-sign \
--key $(SGX_SIGNER_KEY) \
--manifest $< \
--output $@
sudo chown trishita:trishita python3.manifest.sgx python3.sig
python3.sig: python3.manifest.sgx
python3.token: python3.sig
gramine-sgx-get-token \
--output $@ --sig $<
ifeq ($(SGX),)
GRAMINE = gramine-direct
else
GRAMINE = gramine-sgx
endif
encrypt:
gramine-sgx-pf-crypt encrypt -w files/wrap-key -i datasets/cd-sem -o datasets/cd-sem-enc
train-cd: python3.token
sudo $(GRAMINE) $(ENTRYPOINT) train.py \
--dataroot ./datasets/cd-sem-enc/AB \ # this is the path to the encrypted dataset
--name cd_sem_pix2pix_enc \
--model pix2pix \
--direction AtoB \
--gpu_ids -1 \
--display_id 0 \
--num_threads 0 \
--checkpoints_dir ./checkpoints \
--n_epochs 100 \
--gan_mode vanilla \
--beta1 0.5
test-cd: python3.token
sudo $(GRAMINE) $(ENTRYPOINT) test.py \
--dataroot ./datasets/cd-sem/AB \
--checkpoints_dir ./checkpoints \
--name cd_sem_pix2pix \
--model pix2pix \
--direction AtoB \
--gpu_ids -1
Expected results
The model should be able to open the encrypted files transparently
Actual results
warning: encrypted_file_internal_open: pf_open failed: Invalid path
Gramine commit hash
003efc4c8819de47ff11b5a0af7ba09aee7f5fc1
Thank you so much for your help!
When you run this step:
gramine-sgx-pf-crypt encrypt -w files/wrap-key -i datasets/cd-sem -o datasets/cd-sem-enc
The gramine-sgx-pf-crypt tool must say something like Make sure that you put the file under the following path .... Could you send us the output of this tool, with the exact path that it wants you to use?
The Encrypted Files feature of Gramine uses a special format where the file path is part of the verified metadata. So the file path opened at runtime must correspond exactly to the one that was used during encryption of the file using our tool.
Hi!
Thanks for getting back! I changed the encrypt command to:
gramine-sgx-pf-crypt encrypt -w files/wrap-key -i ./datasets/cd-sem -o ./datasets/cd-sem-enc
This directory has a bunch of subdirectories and image files inside of it, and the pf-crypt utility encrypts each of them recursively, and then prints something like:
Gramine's sgx.protected_files must contain this exact path: "./datasets/cd-sem-enc/AB/train/S08_M0308-02MS.tif"
I also zeroed in on the offending line in my ML source code. So apparently the python code tries to run:
sorted(os.walk("./datasets/cd-sem-enc/AB/train")), which should work, but for some reason, it doesn't output any of the files inside of it (and instead we have the gramine warnings of pf_open failing.
You specified 003efc4c8819de47ff11b5a0af7ba09aee7f5fc1 as the Gramine commit hash. I don't see such hash in Gramine. Are you sure that's a correct one? What version of Gramine are you actually using?
Also, could you show more of the Gramine log? The part of the log your copied is a bit too small, I would like to see ~20 lines before this copy-pasted part and ~20 lines after this copy-pasted part.
Hi, apologies for the delay, I've attached the full log here:
[P1:T1:python3] trace: ---- shim_lseek(3, 0x0, SEEK_CUR) = 0x0
[P1:T1:python3] trace: ---- shim_fstat(3, 0x713aa260) = 0x0
[P1:T1:python3] trace: ---- shim_read(3, 0x96fbf80, 0x7f3) ...
[P1:T1:python3] trace: ---- return from shim_read(...) = 0x7f2
[P1:T1:python3] trace: ---- shim_read(3, 0x96fc772, 0x1) ...
[P1:T1:python3] trace: ---- return from shim_read(...) = 0x0
[P1:T1:python3] trace: ---- shim_close(3) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/data/image_folder.py", 0x713a9fd0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/data/__pycache__", 0x713a9b50) = 0x0
warning: Disallowing access to file 'data/__pycache__/image_folder.cpython-38.pyc.158047760'; file is not trusted or allowed.
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "/data/__pycache__/image_folder.cpython-38.pyc.158047760", O_WRONLY|O_CREAT|O_EX
CL|0x80000, 0664) = -13
[P1:T1:python3] trace: ---- shim_stat("./checkpoints/cd_sem_pix2pix_enc", 0x713abe20) = 0x0
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "./checkpoints/cd_sem_pix2pix_enc/train_opt.txt", O_WRONLY|O_CREAT|O_TRUNC|0x800
00, 0666) = 0x3
[P1:T1:python3] trace: ---- shim_fstat(3, 0x713ac050) = 0x0
[P1:T1:python3] trace: ---- shim_ioctl(3, TCGETS, 0x713ac0b0) ...
[P1:T1:python3] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:python3] trace: ---- shim_lseek(3, 0x0, SEEK_CUR) = 0x0
[P1:T1:python3] trace: ---- shim_ioctl(3, TCGETS, 0x713abfd0) ...
[P1:T1:python3] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:python3] trace: ---- shim_lseek(3, 0x0, SEEK_CUR) = 0x0
[P1:T1:python3] trace: ---- shim_write(3, 0x96fbb00, 0xdc2) ...
[P1:T1:python3] trace: ---- return from shim_write(...) = 0xdc2
[P1:T1:python3] trace: ---- shim_close(3) = 0x0
[P1:T1:python3] trace: ---- shim_stat("./datasets/cd-sem-enc/AB/train", 0x713abbd0) = 0x0
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "./datasets/cd-sem-enc/AB/train", O_RDONLY|0x90800, 0000) = 0x3
[P1:T1:python3] trace: ---- shim_newfstatat(3, "", 0x713abaf0, 4096) = 0x0
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
[P1:T1:python3] warning: encrypted_file_internal_open: pf_open failed: Invalid path
...
[P1:T1:python3] trace: ---- shim_getdents64(3, 0x96fbda0, 0x8000) = 0x40
[P1:T1:python3] trace: ---- shim_getdents64(3, 0x96fbda0, 0x8000) = 0x0
[P1:T1:python3] trace: ---- shim_close(3) = 0x0
[P1:T1:python3] trace: ---- shim_write(1, 0x6fdf0120, 0xe3b) ...
# begin print statements
----------------- Options ---------------
batch_size: 1
beta1: 0.5
checkpoints_dir: ./checkpoints
continue_train: False
crop_size: 256
dataroot: ./datasets/cd-sem-enc/AB [default: None]
dataset_mode: aligned
direction: AtoB
display_env: main
display_freq: 400
display_id: 0 [default: 1]
display_ncols: 4
display_port: 8097
display_server: http://localhost
display_winsize: 256
epoch: latest
epoch_count: 1
gan_mode: vanilla
gpu_ids: -1 [default: 0]
init_gain: 0.02
init_type: normal
OS WALK
./datasets/cd-sem-enc/AB/train
[('./datasets/cd-sem-enc/AB/train', [], [])]
dataset [AlignedDataset] was created
# end print statements
[P1:T1:python3] trace: ---- return from shim_write(...) = 0xe3b
[P1:T1:python3] trace: ---- shim_stat("/", 0x713ab4c0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/usr/lib/python3.8", 0x713ab4c0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/usr/lib/python3.8/lib-dynload", 0x713ab4c0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/usr/local/lib/python3.8/dist-packages", 0x713ab4c0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/usr/lib/python3/dist-packages", 0x713ab4c0) = 0x0
[P1:T1:python3] trace: ---- shim_stat("/usr/lib/python3/dist-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so", 0x713ab1a0) = 0x0
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/python3/dist-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:python3] trace: ---- shim_read(3, 0x713aae28, 0x340) ...
[P1:T1:python3] trace: ---- return from shim_read(...) = 0x340
[P1:T1:python3] trace: ---- shim_newfstatat(3, "", 0x713aacc0, 4096) = 0x0
[P1:T1:python3] trace: ---- shim_mmap(0, 0x582d0, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:python3] trace: ---- return from shim_mmap(...) = 0x8fad000
[P1:T1:python3] trace: ---- shim_mmap(0x8fc3000, 0x20000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) ...
[P1:T1:python3] trace: ---- return from shim_mmap(...) = 0x8fc3000
[P1:T1:python3] trace: ---- shim_mmap(0x8fe3000, 0x18000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x36000) ...
[P1:T1:python3] trace: ---- return from shim_mmap(...) = 0x8fe3000
[P1:T1:python3] trace: ---- shim_mmap(0x8ffb000, 0xb000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4d000) ...
[P1:T1:python3] trace: ---- return from shim_mmap(...) = 0x8fe3000
[P1:T1:python3] trace: ---- shim_mmap(0x8ffb000, 0xb000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4d000) ...
[P1:T1:python3] trace: ---- return from shim_mmap(...) = 0x8ffb000
[P1:T1:python3] trace: ---- shim_close(3) = 0x0
[P1:T1:python3] debug: glibc register library /usr/lib/python3/dist-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so loaded at 0x08fad000
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "/lib/libapt-pkg.so.6.0", O_RDONLY|0x80000, 0000) = -2
[P1:T1:python3] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libapt-pkg.so.6.0", O_RDONLY|0x80000, 0000) = 0x3
The commit hash is the latest commit I believe. I'm using the current version on the master branch Thanks!
Here are also the mount outputs: (the /datasets/cd-sem-enc/ is the encrypted drive that I am unable to access)
debug: mounting "file:." (chroot) under /
debug: mounting "proc" (pseudo) under /proc
debug: mounting "dev" (pseudo) under /dev
debug: mounting "dev:tty" (chroot) under /dev/tty
debug: mounting "sys" (pseudo) under /sys
[P1:T1:] debug: mounting "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc" (chroot) under /lib
[P1:T1:] debug: mounting "file:/lib/x86_64-linux-gnu" (chroot) under /lib/x86_64-linux-gnu
[P1:T1:] debug: mounting "file:./datasets/cd-sem-enc/" (encrypted) under /datasets/cd-sem-enc/
[P1:T1:] debug: mounting "file:/etc" (chroot) under /etc
[P1:T1:] debug: mounting "file:/usr" (chroot) under /usr
[P1:T1:] debug: mounting "file:/tmp" (chroot) under /tmp
[P1:T1:] debug: mounting "file:/usr/lib" (chroot) under /usr/lib
I looked at gramine-sgx-pf-crypt's output:
Encrypting: ./datasets/cd-sem/AB/train/S08_M0308-02MS.tif -> ./datasets/cd-sem-enc/AB/train/S08_M0308-02MS.tif
(Gramine's sgx.protected_files must contain this exact path: "./datasets/cd-sem-enc/AB/train/S08_M0308-02MS.tif")
I even tried opening just this single file by using the exact path:
with open("./datasets/cd-sem-enc/AB/train/S08_M0308-02MS.tif", "rb") as fdesc:
print(fdesc.read())
and I still get the same error message
Perhaps you can change the mount line in the manifest to:
{ path = "/datasets/cd-sem-enc", uri = "file:./datasets/cd-sem-enc", type = "encrypted"},
then it should work.
The commit hash is the latest commit I believe. I'm using the current version on the master branch
I don't think this is true. There is no such commit hash in Gramine. What does cd gramine/ && git log | head --lines 20 show you?
Notice that we fixed the bug with Encrypted Files just two weeks ago: https://github.com/gramineproject/gramine/commit/c92978c7645f89bd01a4dec09d2ab7eb33dbf840. This bug looks like something that may explain your issues. Please update to the latest master branch and try again.
Ah, you are right -- sorry, my mistake. This seems to not be the latest commit:
commit 7871759db17621b0ca4db5d39b1f275256c98348
Author: Borys Popławski <[email protected]>
Date: Thu Jun 23 22:57:42 2022 +0200
[LibOS] Rename syscalls from `shim_do_x` to `libos_syscall_x`
This is first part of the big `shim` -> `libos` rename.
Signed-off-by: Borys Popławski <[email protected]>
commit 2cb071b576b70bc26e2b3c4211bc00b3ec534083
Author: Borys Popławski <[email protected]>
Date: Tue Jun 21 20:33:03 2022 +0200
[LibOS] Add `mask` to `setflags` callback in `struct shim_fs_ops`
This allows to change only a subset of flags, e.g. just `O_NONBLOCK`.
Additionally this commit enables setting `O_NONBLOCK` on UNIX sockets
that were not yet bound or connected.
Let me reinstall and try it out
Perhaps you can change the mount line in the manifest to:
{ path = "/datasets/cd-sem-enc", uri = "file:./datasets/cd-sem-enc", type = "encrypted"},then it should work.
Unfortunately that doesn't seem to work, but I'm trying it out with the latest version and lets see.
The commit hash is the latest commit I believe. I'm using the current version on the master branch
I don't think this is true. There is no such commit hash in Gramine. What does
cd gramine/ && git log | head --lines 20show you?Notice that we fixed the bug with Encrypted Files just two weeks ago: c92978c. This bug looks like something that may explain your issues. Please update to the latest master branch and try again.
Ok, so I tried again with the latest gramine build:
commit dcc0eb5c6b09e8f3703805923af6d30a72a7af7d
Author: Michał Kowalczyk <[email protected]>
Date: Mon Sep 26 12:06:31 2022 +0200
Bump Gramine version to 1.3post-UNRELEASED
Signed-off-by: Michał Kowalczyk <[email protected]>
commit a6887a5321433c8605bdbecea9f3d45afed66993
Author: Michał Kowalczyk <[email protected]>
Date: Mon Sep 26 12:05:48 2022 +0200
Bump Gramine version to 1.3
Signed-off-by: Michał Kowalczyk <[email protected]>
commit 4f259b80ed4c40e7cccde70e822d378754dacb97
Author: Borys Popławski <[email protected]>
Date: Thu Sep 22 13:18:15 2022 +0200
But I'm getting the exact same error there as well :(
I just tested it. It seems you need to change the encryption command to:
gramine-sgx-pf-crypt encrypt -w test_key -i datasets/cd-sem -o datasets/cd-sem-enc
The leading './' is causing the mismatch (at least in my test).
@lejunzhu Thanks! I know what the problem is. We don't normalize the paths in the gramine-sgx-pf-crypt tool. Let me cook a PR that fixes that...
@fork-while-1 @lejunzhu If you have time, please check https://github.com/gramineproject/gramine/pull/937. It should fix the error.
@fork-while-1 @lejunzhu If you have time, please check #937. It should fix the error.
Yes, it fixes the issue in my test.